2024-02-16T22:00:43.482 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-16T22:00:43.482 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_pdonnell@teuthology --name pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728 --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728/orig.config.yaml 2024-02-16T22:00:43.535 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-16T22:00:43.614 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728 branch: wip-batrick-testing-20240215.160715 description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7562728' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240215.160715 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: f78a58c0ffd401d1493058a1022c35f011d65275 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f78a58c0ffd401d1493058a1022c35f011d65275 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: i64440 sha1: ed7ab725e1a43615009dfb74a42bf33d80081bc8 owner: scheduled_pdonnell@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2634 sha1: f78a58c0ffd401d1493058a1022c35f011d65275 sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: i64440 suite_path: /home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: ed7ab725e1a43615009dfb74a42bf33d80081bc8 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRbrMcmK/Fgj1sNYHecH4R601mYPfWJcYpCFgAAMGA666vdznFRd/r8YlTThq7pFdB15gSrgdBTUFpbfpr8TKs= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLvBv5Qb2+qoNasbzTBfm9CW43nFJ3AuTGtkd9UtTxA6WnL9GCNIwXZoZuiZ+0OSKY6vCiHHRNCdmkPaC+xBKGs= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - full_sequential_finally: - duration: 3600 sleep: null - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/pdonnell/src/github.com_batrick_ceph_i64440/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-16_18:26:20 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-16T22:00:43.614 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa; will attempt to use it 2024-02-16T22:00:43.615 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks 2024-02-16T22:00:43.615 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-16T22:00:43.616 INFO:teuthology.task.internal:Checking packages... 2024-02-16T22:00:43.637 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f78a58c0ffd401d1493058a1022c35f011d65275' 2024-02-16T22:00:43.637 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-16T22:00:43.637 INFO:teuthology.packaging:ref: None 2024-02-16T22:00:43.637 INFO:teuthology.packaging:tag: None 2024-02-16T22:00:43.638 INFO:teuthology.packaging:branch: wip-batrick-testing-20240215.160715 2024-02-16T22:00:43.638 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:00:43.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240215.160715 2024-02-16T22:00:43.929 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1450.gf78a58c0 2024-02-16T22:00:43.931 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-16T22:00:43.937 INFO:teuthology.task.internal:no buildpackages task found 2024-02-16T22:00:43.937 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-16T22:00:43.980 INFO:teuthology.task.internal:Saving configuration 2024-02-16T22:00:43.999 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-16T22:00:44.005 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-16T22:00:44.025 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-16 21:55:18.217325', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRbrMcmK/Fgj1sNYHecH4R601mYPfWJcYpCFgAAMGA666vdznFRd/r8YlTThq7pFdB15gSrgdBTUFpbfpr8TKs='} 2024-02-16T22:00:44.049 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-16 21:55:18.219281', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLvBv5Qb2+qoNasbzTBfm9CW43nFJ3AuTGtkd9UtTxA6WnL9GCNIwXZoZuiZ+0OSKY6vCiHHRNCdmkPaC+xBKGs='} 2024-02-16T22:00:44.049 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-16T22:00:44.056 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-16T22:00:44.056 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-16T22:00:44.056 INFO:teuthology.run_tasks:Running task console_log... 2024-02-16T22:00:44.135 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0ffb2ce0d0>, signals=[15]) 2024-02-16T22:00:44.135 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-16T22:00:44.142 INFO:teuthology.task.internal:Opening connections... 2024-02-16T22:00:44.142 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:00:44.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:00:44.247 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-02-16T22:00:44.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:00:44.335 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-16T22:00:44.342 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2024-02-16T22:00:44.360 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2024-02-16T22:00:44.361 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Stream" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="9" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="9" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el9" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:LOGO="fedora-logo-icon" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2024-02-16T22:00:44.418 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-16T22:00:44.419 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-16T22:00:44.419 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-16T22:00:44.419 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2024-02-16T22:00:44.443 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-02-16T22:00:44.460 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-02-16T22:00:44.461 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-02-16T22:00:44.516 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-16T22:00:44.517 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-16T22:00:44.518 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-02-16T22:00:44.543 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-16T22:00:44.553 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-16T22:00:44.579 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-16T22:00:44.580 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2024-02-16T22:00:44.582 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-02-16T22:00:44.598 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-16T22:00:44.603 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-16T22:00:44.603 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2024-02-16T22:00:44.642 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-02-16T22:00:44.971 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-16T22:00:45.031 INFO:teuthology.run_tasks:Running task kernel... 2024-02-16T22:00:45.053 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-16T22:00:45.053 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-16T22:00:45.054 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-16T22:00:45.054 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-16T22:00:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-16T22:00:45.072 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-02-16T22:00:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-16T22:00:45.073 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-02-16T22:00:45.130 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-361.el9.x86_64 2024-02-16T22:00:45.130 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-361.el9.x86_64 2024-02-16T22:00:45.130 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-02-16T22:00:45.135 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-361.el9.x86_64 2024-02-16T22:00:45.136 INFO:teuthology.task.kernel:Running kernel on smithi002: 5.14.0-361.el9.x86_64 2024-02-16T22:00:45.136 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum install -y kernel 2024-02-16T22:00:47.377 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-02-16T22:00:47.955 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 6.2 MB/s | 6.7 MB 00:01 2024-02-16T22:00:48.868 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-02-16T22:00:49.322 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-02-16T22:00:51.442 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-02-16T22:00:51.504 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-02-16T22:00:54.574 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-02-16T22:00:59.439 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.1 MB 00:01 2024-02-16T22:01:00.882 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2024-02-16T22:01:07.850 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 3.0 MB/s | 20 MB 00:06 2024-02-16T22:01:12.825 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-02-16T22:01:15.123 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-16T22:01:15.123 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-16T22:01:15.212 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:01:15.219 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:01:15.219 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:01:15.219 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-02-16T22:01:15.220 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:01:15.221 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 100 M 2024-02-16T22:01:15.221 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 123 M 2024-02-16T22:01:15.221 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:01:15.734 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 12 MB/s | 6.1 MB 00:00 2024-02-16T22:01:16.185 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-02-16T22:01:16.943 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 23 MB/s | 39 MB 00:01 2024-02-16T22:01:17.477 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): kernel-modules-core-5.14.0-421.el9.x86_6 19 MB/s | 33 MB 00:01 2024-02-16T22:01:17.480 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:01:17.480 INFO:teuthology.orchestra.run.smithi103.stdout:Total 44 MB/s | 100 MB 00:02 2024-02-16T22:01:17.480 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:01:17.848 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:01:17.848 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:01:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 761 kB/s | 19 MB 00:25 2024-02-16T22:01:18.841 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:01:18.841 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:01:21.248 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:01:22.006 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-16T22:01:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-16T22:01:23.574 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-02-16T22:01:24.897 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-16T22:01:25.058 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 15 kB 00:00 2024-02-16T22:01:25.083 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-16T22:01:26.145 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 21 MB/s | 20 MB 00:00 2024-02-16T22:01:29.803 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-16T22:01:29.913 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:01:31.151 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-02-16T22:01:33.541 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-16T22:01:33.541 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-16T22:01:33.639 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:01:33.647 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:01:33.647 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:01:33.647 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:01:33.647 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:01:33.647 INFO:teuthology.orchestra.run.smithi002.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-16T22:01:33.647 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-16T22:01:33.647 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout:Install 4 Packages 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 100 M 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 123 M 2024-02-16T22:01:33.648 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:01:34.377 INFO:teuthology.orchestra.run.smithi002.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 8.3 MB/s | 6.1 MB 00:00 2024-02-16T22:01:34.661 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:01:35.252 INFO:teuthology.orchestra.run.smithi002.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 13 MB/s | 21 MB 00:01 2024-02-16T22:01:36.402 INFO:teuthology.orchestra.run.smithi002.stdout:(3/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 14 MB/s | 39 MB 00:02 2024-02-16T22:01:36.728 INFO:teuthology.orchestra.run.smithi002.stdout:(4/4): kernel-modules-core-5.14.0-421.el9.x86_6 14 MB/s | 33 MB 00:02 2024-02-16T22:01:36.731 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:01:36.732 INFO:teuthology.orchestra.run.smithi002.stdout:Total 32 MB/s | 100 MB 00:03 2024-02-16T22:01:36.732 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:01:37.194 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:01:37.194 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:01:38.275 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:01:38.276 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:01:40.796 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:01:41.486 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-16T22:01:41.628 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-16T22:01:44.506 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-16T22:01:44.720 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-16T22:01:49.785 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-16T22:01:49.894 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:01:54.992 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:02:04.053 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:02:04.054 INFO:teuthology.orchestra.run.smithi103.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-16T22:02:04.054 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:02:04.125 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:02:04.533 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:02:04.533 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-16T22:02:04.533 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-16T22:02:04.533 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-16T22:02:04.959 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:02:04.959 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:02:04.959 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:02:04.959 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-16T22:02:04.959 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-16T22:02:04.959 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-16T22:02:04.959 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-16T22:02:04.959 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:02:04.959 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:02:05.464 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-02-16T22:02:05.652 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-16T22:02:05.653 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-16T22:02:05.653 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-16T22:02:05.653 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-16T22:02:05.653 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-16T22:02:05.653 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-16T22:02:05.653 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-02-16T22:02:05.653 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-02-16T22:02:05.760 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-16T22:02:05.761 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-16T22:02:05.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-02-16T22:02:05.803 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-02-16T22:02:05.803 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 68.el9 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8264254 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-16T22:02:05.804 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:02:05.805 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-02-16T22:02:05.806 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-16T22:02:05.806 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 5.14.0-421.el9.x86_64 2024-02-16T22:02:05.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-16T22:02:06.415 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-02-16T22:02:07.796 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-16T22:02:07.830 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-02-16T22:02:07.832 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-02-16T22:02:07.847 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.BEdnV1eIQe 2024-02-16T22:02:07.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BEdnV1eIQe 2024-02-16T22:02:07.916 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.BEdnV1eIQe 2024-02-16T22:02:08.053 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.BEdnV1eIQe is 6KB 2024-02-16T22:02:08.103 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.BEdnV1eIQe 2024-02-16T22:02:08.118 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-02-16T22:02:08.183 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-16T22:02:08.183 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-16T22:02:08.183 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-16T22:02:08.183 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-16T22:02:08.184 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-16T22:02:08.334 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-02-16T22:02:25.631 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:02:25.631 INFO:teuthology.orchestra.run.smithi002.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-16T22:02:25.631 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:02:25.704 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:02:26.100 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:02:26.101 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-16T22:02:26.101 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-16T22:02:26.101 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-16T22:02:26.514 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-16T22:02:26.514 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:02:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:02:26.515 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-16T22:02:26.515 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-16T22:02:26.515 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-16T22:02:26.515 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-16T22:02:26.515 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:02:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:02:26.981 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-02-16T22:02:27.121 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-16T22:02:27.121 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-16T22:02:27.121 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-16T22:02:27.121 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-16T22:02:27.122 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-16T22:02:27.122 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-16T22:02:27.122 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2024-02-16T22:02:27.122 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-02-16T22:02:27.235 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-16T22:02:27.235 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-16T22:02:27.235 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.06 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 68.el9 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:Group : Unspecified 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 8264254 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2024-02-16T22:02:27.284 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : builder@centos.org 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : CentOS 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-16T22:02:27.285 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-16T22:02:27.286 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2024-02-16T22:02:27.286 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:02:27.286 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2024-02-16T22:02:27.287 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-16T22:02:27.287 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 5.14.0-421.el9.x86_64 2024-02-16T22:02:27.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-16T22:02:27.804 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2024-02-16T22:02:29.221 INFO:teuthology.orchestra.run.smithi002.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-16T22:02:29.252 INFO:teuthology.orchestra.run.smithi002.stderr:done 2024-02-16T22:02:29.254 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-02-16T22:02:29.270 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.PGHJcVie53 2024-02-16T22:02:29.270 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PGHJcVie53 2024-02-16T22:02:29.358 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.PGHJcVie53 2024-02-16T22:02:29.464 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.PGHJcVie53 is 6KB 2024-02-16T22:02:29.512 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.PGHJcVie53 2024-02-16T22:02:29.527 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2024-02-16T22:02:29.603 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-16T22:02:29.603 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-16T22:02:29.603 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-16T22:02:29.603 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-16T22:02:29.605 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-16T22:02:29.745 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2024-02-16T22:02:38.344 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-16T22:02:38.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:02:38.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:02:56.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:02:59.748 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-16T22:02:59.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-16T22:02:59.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:03:05.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:03:05.705 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:03:08.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:03:20.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:03:20.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:03:23.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:03:38.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:03:38.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:03:59.772 DEBUG:teuthology.orchestra.remote:timed out 2024-02-16T22:04:08.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-16T22:04:08.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:04:09.120 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-02-16T22:04:09.564 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-16T22:04:09.565 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-421.el9.x86_64"... 2024-02-16T22:04:09.565 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-02-16T22:04:09.578 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-421.el9.x86_64 2024-02-16T22:04:09.579 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-16T22:04:09.579 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-16T22:04:09.579 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-16T22:04:10.580 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-16T22:04:10.580 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-16T22:04:10.631 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2024-02-16T22:04:10.667 DEBUG:teuthology.parallel:result is None 2024-02-16T22:04:10.875 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-02-16T22:04:11.236 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:04:11.236 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-421.el9.x86_64"... 2024-02-16T22:04:11.236 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-02-16T22:04:11.250 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-421.el9.x86_64 2024-02-16T22:04:11.250 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-16T22:04:11.250 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-16T22:04:11.251 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-16T22:04:12.252 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-16T22:04:12.252 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-16T22:04:12.326 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-02-16T22:04:12.368 DEBUG:teuthology.parallel:result is None 2024-02-16T22:04:12.369 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-16T22:04:12.376 INFO:teuthology.task.internal:Creating test directory... 2024-02-16T22:04:12.376 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-16T22:04:12.378 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-16T22:04:12.397 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-16T22:04:12.413 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-16T22:04:12.447 INFO:teuthology.task.internal:Creating archive directory... 2024-02-16T22:04:12.447 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-16T22:04:12.449 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-16T22:04:12.490 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-16T22:04:12.497 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-16T22:04:12.497 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-16T22:04:12.508 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-16T22:04:12.536 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-16T22:04:12.549 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-16T22:04:12.561 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-16T22:04:12.574 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-16T22:04:12.576 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-16T22:04:12.583 INFO:teuthology.task.internal:Configuring sudo... 2024-02-16T22:04:12.583 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-16T22:04:12.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-16T22:04:12.649 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-16T22:04:12.658 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-16T22:04:12.659 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-16T22:04:12.666 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-16T22:04:12.708 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-16T22:04:12.764 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-16T22:04:12.817 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-16T22:04:12.903 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-16T22:04:12.949 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:04:12.949 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-16T22:04:13.019 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-16T22:04:13.065 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-16T22:04:13.113 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-16T22:04:13.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-16T22:04:13.240 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:04:13.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-16T22:04:13.309 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2024-02-16T22:04:13.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-02-16T22:04:13.354 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-16T22:04:13.395 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-16T22:04:13.921 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-16T22:04:13.929 INFO:teuthology.task.internal:Starting timer... 2024-02-16T22:04:13.929 INFO:teuthology.run_tasks:Running task pcp... 2024-02-16T22:04:13.966 INFO:teuthology.run_tasks:Running task selinux... 2024-02-16T22:04:13.998 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-16T22:04:13.999 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2024-02-16T22:04:14.096 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2024-02-16T22:04:14.097 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-02-16T22:04:14.163 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-02-16T22:04:14.165 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-16T22:04:14.165 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2024-02-16T22:04:14.184 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2024-02-16T22:04:14.185 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-02-16T22:04:14.211 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-02-16T22:04:14.211 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-02-16T22:04:14.212 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-16T22:04:14.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-16T22:04:14.256 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-16T22:04:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-16T22:04:14.285 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-16T22:04:14.285 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2024-02-16T22:04:14.395 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-02-16T22:04:14.429 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-16T22:04:14.438 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-16T22:04:14.455 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-16T22:04:14.457 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-16T22:08:27.083 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-02-16T22:08:27.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-16T22:08:27.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:08:27.172 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-02-16T22:08:27.250 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-02-16T22:08:27.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:08:27.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:08:27.337 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-02-16T22:08:27.414 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:08:27.415 INFO:teuthology.run_tasks:Running task clock... 2024-02-16T22:08:27.425 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-16T22:08:27.425 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-16T22:08:27.426 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-16T22:08:27.428 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-16T22:08:27.428 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-16T22:08:27.458 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-16T22:08:27.475 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-16T22:08:27.486 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-16T22:08:27.503 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-16T22:08:27.504 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2024-02-16T22:08:27.518 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2024-02-16T22:08:27.533 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-02-16T22:08:27.534 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-16T22:08:27.546 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-02-16T22:08:27.551 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-16T22:08:27.562 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-16T22:08:27.579 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-16T22:08:27.602 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-02-16T22:08:27.605 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-16T22:08:27.605 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-02-16T22:08:27.605 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-16T22:08:27.605 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-16T22:08:27.605 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-16T22:08:27.606 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-16T22:08:27.637 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-02-16T22:08:27.641 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-16T22:08:27.641 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-02-16T22:08:27.641 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-16T22:08:27.641 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-16T22:08:27.641 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-16T22:08:27.641 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-16T22:08:27.641 INFO:teuthology.run_tasks:Running task install... 2024-02-16T22:08:27.649 DEBUG:teuthology.task.install:project ceph 2024-02-16T22:08:27.649 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'}} 2024-02-16T22:08:27.649 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'} 2024-02-16T22:08:27.649 INFO:teuthology.task.install:Using flavor: default 2024-02-16T22:08:27.654 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-16T22:08:27.654 INFO:teuthology.task.install:extra packages: [] 2024-02-16T22:08:27.655 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'tag': None, 'wait_for_package': False} 2024-02-16T22:08:27.655 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-16T22:08:27.655 INFO:teuthology.packaging:ref: None 2024-02-16T22:08:27.655 INFO:teuthology.packaging:tag: None 2024-02-16T22:08:27.655 INFO:teuthology.packaging:branch: quincy 2024-02-16T22:08:27.655 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:08:27.655 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-16T22:08:27.657 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'tag': None, 'wait_for_package': False} 2024-02-16T22:08:27.657 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-16T22:08:27.657 INFO:teuthology.packaging:ref: None 2024-02-16T22:08:27.657 INFO:teuthology.packaging:tag: None 2024-02-16T22:08:27.657 INFO:teuthology.packaging:branch: quincy 2024-02-16T22:08:27.657 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:08:27.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-16T22:08:28.001 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/b88fae66e6ad896087939a6bde358f026d85f561/centos/9/flavors/default/ 2024-02-16T22:08:28.001 INFO:teuthology.task.install.rpm:Package version is 17.2.7-513.gb88fae66 2024-02-16T22:08:28.018 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/b88fae66e6ad896087939a6bde358f026d85f561/centos/9/flavors/default/ 2024-02-16T22:08:28.018 INFO:teuthology.task.install.rpm:Package version is 17.2.7-513.gb88fae66 2024-02-16T22:08:28.111 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/b88fae66e6ad896087939a6bde358f026d85f561/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/b88fae66e6ad896087939a6bde358f026d85f561/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/b88fae66e6ad896087939a6bde358f026d85f561/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-16T22:08:28.112 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:08:28.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-16T22:08:28.140 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-16T22:08:28.140 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-16T22:08:28.140 INFO:teuthology.packaging:ref: None 2024-02-16T22:08:28.140 INFO:teuthology.packaging:tag: None 2024-02-16T22:08:28.140 INFO:teuthology.packaging:branch: quincy 2024-02-16T22:08:28.140 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:08:28.140 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-16T22:08:28.193 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/b88fae66e6ad896087939a6bde358f026d85f561/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/b88fae66e6ad896087939a6bde358f026d85f561/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/b88fae66e6ad896087939a6bde358f026d85f561/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-16T22:08:28.193 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:08:28.193 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-16T22:08:28.207 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-16T22:08:28.223 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-16T22:08:28.223 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-16T22:08:28.223 INFO:teuthology.packaging:ref: None 2024-02-16T22:08:28.223 INFO:teuthology.packaging:tag: None 2024-02-16T22:08:28.223 INFO:teuthology.packaging:branch: quincy 2024-02-16T22:08:28.223 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:08:28.223 DEBUG:teuthology.orchestra.run.smithi002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-16T22:08:28.286 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-16T22:08:28.294 DEBUG:teuthology.orchestra.run.smithi002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-16T22:08:28.353 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-02-16T22:08:28.355 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-02-16T22:08:28.377 DEBUG:teuthology.orchestra.run.smithi002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-16T22:08:28.445 INFO:teuthology.orchestra.run.smithi002.stdout:check_obsoletes = 1 2024-02-16T22:08:28.447 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum clean all 2024-02-16T22:08:28.608 INFO:teuthology.orchestra.run.smithi103.stdout:74 files removed 2024-02-16T22:08:28.634 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw 2024-02-16T22:08:28.698 INFO:teuthology.orchestra.run.smithi002.stdout:74 files removed 2024-02-16T22:08:28.725 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-radosgw 2024-02-16T22:08:29.531 INFO:teuthology.orchestra.run.smithi002.stdout:ceph packages for x86_64 160 kB/s | 77 kB 00:00 2024-02-16T22:08:29.547 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 130 kB/s | 77 kB 00:00 2024-02-16T22:08:29.856 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 35 kB/s | 9.1 kB 00:00 2024-02-16T22:08:29.973 INFO:teuthology.orchestra.run.smithi002.stdout:ceph noarch packages 23 kB/s | 9.1 kB 00:00 2024-02-16T22:08:30.382 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 3.7 kB/s | 1.8 kB 00:00 2024-02-16T22:08:30.532 INFO:teuthology.orchestra.run.smithi002.stdout:ceph source packages 3.5 kB/s | 1.8 kB 00:00 2024-02-16T22:08:30.951 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-16T22:08:31.034 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-02-16T22:08:32.418 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-02-16T22:08:32.510 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-02-16T22:08:33.919 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-02-16T22:08:34.719 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-02-16T22:08:41.734 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 19 MB 00:05 2024-02-16T22:08:42.667 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 19 MB 00:07 2024-02-16T22:08:46.951 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.1 MB 00:01 2024-02-16T22:08:47.901 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-02-16T22:08:48.619 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-02-16T22:08:49.444 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2024-02-16T22:08:50.115 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-02-16T22:08:50.714 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2024-02-16T22:08:55.037 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-02-16T22:08:55.787 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-02-16T22:08:57.668 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:08:57.676 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:08:57.676 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:17.2.7-513.gb88fae66.el9 ceph 14 M 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 3.1 M 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 2.9 M 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:17.2.7-513.gb88fae66.el9 ceph 5.0 M 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:17.2.7-513.gb88fae66.el9 ceph 23 M 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:17.2.7-513.gb88fae66.el9 ceph 25 k 2024-02-16T22:08:57.677 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 635 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 458 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 4.9 M 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:17.2.7-513.gb88fae66.el9 ceph 45 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:17.2.7-513.gb88fae66.el9 ceph 116 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:17.2.7-513.gb88fae66.el9 ceph 156 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-16T22:08:57.678 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:17.2.7-513.gb88fae66.el9 ceph 320 k 2024-02-16T22:08:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:17.2.7-513.gb88fae66.el9 ceph 293 k 2024-02-16T22:08:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:17.2.7-513.gb88fae66.el9 ceph 99 k 2024-02-16T22:08:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-16T22:08:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:08:57.679 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:08:57.679 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:08:57.679 INFO:teuthology.orchestra.run.smithi103.stdout:Install 22 Packages 2024-02-16T22:08:57.679 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 2 Packages 2024-02-16T22:08:57.679 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:08:57.680 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 58 M 2024-02-16T22:08:57.680 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:08:58.514 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:08:58.528 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:08:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:08:58.528 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:08:58.528 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:08:58.528 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw x86_64 2:17.2.7-513.gb88fae66.el9 ceph 14 M 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 3.1 M 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 2.9 M 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base x86_64 2:17.2.7-513.gb88fae66.el9 ceph 5.0 M 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common x86_64 2:17.2.7-513.gb88fae66.el9 ceph 23 M 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux x86_64 2:17.2.7-513.gb88fae66.el9 ceph 25 k 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 635 k 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-16T22:08:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 458 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2 x86_64 2:17.2.7-513.gb88fae66.el9 ceph 4.9 M 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse x86_64 2:17.2.7-513.gb88fae66.el9 ceph 45 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common x86_64 2:17.2.7-513.gb88fae66.el9 ceph 116 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs x86_64 2:17.2.7-513.gb88fae66.el9 ceph 156 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados x86_64 2:17.2.7-513.gb88fae66.el9 ceph 320 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd x86_64 2:17.2.7-513.gb88fae66.el9 ceph 293 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw x86_64 2:17.2.7-513.gb88fae66.el9 ceph 99 k 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-16T22:08:58.530 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:08:58.531 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:08:58.531 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:08:58.531 INFO:teuthology.orchestra.run.smithi002.stdout:Install 22 Packages 2024-02-16T22:08:58.531 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 2 Packages 2024-02-16T22:08:58.531 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:08:58.531 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 58 M 2024-02-16T22:08:58.531 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:08:58.755 INFO:teuthology.orchestra.run.smithi103.stdout:(1/24): ceph-radosgw-17.2.7-513.gb88fae66.el9.x 14 MB/s | 14 MB 00:01 2024-02-16T22:08:58.796 INFO:teuthology.orchestra.run.smithi103.stdout:(2/24): ceph-selinux-17.2.7-513.gb88fae66.el9.x 610 kB/s | 25 kB 00:00 2024-02-16T22:08:58.863 INFO:teuthology.orchestra.run.smithi103.stdout:(3/24): libcephfs2-17.2.7-513.gb88fae66.el9.x86 9.3 MB/s | 635 kB 00:00 2024-02-16T22:08:58.964 INFO:teuthology.orchestra.run.smithi103.stdout:(4/24): libradosstriper1-17.2.7-513.gb88fae66.e 4.5 MB/s | 458 kB 00:00 2024-02-16T22:08:59.739 INFO:teuthology.orchestra.run.smithi103.stdout:(5/24): librgw2-17.2.7-513.gb88fae66.el9.x86_64 6.3 MB/s | 4.9 MB 00:00 2024-02-16T22:08:59.823 INFO:teuthology.orchestra.run.smithi103.stdout:(6/24): python3-ceph-argparse-17.2.7-513.gb88fa 541 kB/s | 45 kB 00:00 2024-02-16T22:08:59.915 INFO:teuthology.orchestra.run.smithi103.stdout:(7/24): python3-ceph-common-17.2.7-513.gb88fae6 1.2 MB/s | 116 kB 00:00 2024-02-16T22:08:59.998 INFO:teuthology.orchestra.run.smithi103.stdout:(8/24): python3-cephfs-17.2.7-513.gb88fae66.el9 1.8 MB/s | 156 kB 00:00 2024-02-16T22:09:00.012 INFO:teuthology.orchestra.run.smithi002.stdout:(1/24): ceph-base-17.2.7-513.gb88fae66.el9.x86_ 3.5 MB/s | 5.0 MB 00:01 2024-02-16T22:09:00.074 INFO:teuthology.orchestra.run.smithi103.stdout:(9/24): python3-rados-17.2.7-513.gb88fae66.el9. 4.2 MB/s | 320 kB 00:00 2024-02-16T22:09:00.096 INFO:teuthology.orchestra.run.smithi002.stdout:(2/24): ceph-selinux-17.2.7-513.gb88fae66.el9.x 302 kB/s | 25 kB 00:00 2024-02-16T22:09:00.166 INFO:teuthology.orchestra.run.smithi103.stdout:(10/24): python3-rbd-17.2.7-513.gb88fae66.el9.x 3.1 MB/s | 293 kB 00:00 2024-02-16T22:09:00.258 INFO:teuthology.orchestra.run.smithi103.stdout:(11/24): python3-rgw-17.2.7-513.gb88fae66.el9.x 1.1 MB/s | 99 kB 00:00 2024-02-16T22:09:00.321 INFO:teuthology.orchestra.run.smithi002.stdout:(3/24): libcephfs2-17.2.7-513.gb88fae66.el9.x86 2.8 MB/s | 635 kB 00:00 2024-02-16T22:09:00.400 INFO:teuthology.orchestra.run.smithi103.stdout:(12/24): ceph-base-17.2.7-513.gb88fae66.el9.x86 1.9 MB/s | 5.0 MB 00:02 2024-02-16T22:09:00.425 INFO:teuthology.orchestra.run.smithi103.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 954 kB/s | 159 kB 00:00 2024-02-16T22:09:00.441 INFO:teuthology.orchestra.run.smithi103.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-16T22:09:00.480 INFO:teuthology.orchestra.run.smithi002.stdout:(4/24): libradosstriper1-17.2.7-513.gb88fae66.e 2.8 MB/s | 458 kB 00:00 2024-02-16T22:09:00.484 INFO:teuthology.orchestra.run.smithi103.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-02-16T22:09:00.509 INFO:teuthology.orchestra.run.smithi103.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-02-16T22:09:00.526 INFO:teuthology.orchestra.run.smithi103.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 995 kB/s | 41 kB 00:00 2024-02-16T22:09:01.101 INFO:teuthology.orchestra.run.smithi103.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 84 kB/s | 49 kB 00:00 2024-02-16T22:09:01.302 INFO:teuthology.orchestra.run.smithi103.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 336 kB/s | 67 kB 00:00 2024-02-16T22:09:01.327 INFO:teuthology.orchestra.run.smithi103.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 378 kB/s | 309 kB 00:00 2024-02-16T22:09:01.730 INFO:teuthology.orchestra.run.smithi002.stdout:(5/24): ceph-radosgw-17.2.7-513.gb88fae66.el9.x 4.5 MB/s | 14 MB 00:03 2024-02-16T22:09:01.797 INFO:teuthology.orchestra.run.smithi002.stdout:(6/24): librgw2-17.2.7-513.gb88fae66.el9.x86_64 3.7 MB/s | 4.9 MB 00:01 2024-02-16T22:09:01.822 INFO:teuthology.orchestra.run.smithi002.stdout:(7/24): python3-ceph-argparse-17.2.7-513.gb88fa 492 kB/s | 45 kB 00:00 2024-02-16T22:09:01.872 INFO:teuthology.orchestra.run.smithi002.stdout:(8/24): python3-ceph-common-17.2.7-513.gb88fae6 1.5 MB/s | 116 kB 00:00 2024-02-16T22:09:01.898 INFO:teuthology.orchestra.run.smithi002.stdout:(9/24): python3-cephfs-17.2.7-513.gb88fae66.el9 2.0 MB/s | 156 kB 00:00 2024-02-16T22:09:01.957 INFO:teuthology.orchestra.run.smithi002.stdout:(10/24): python3-rados-17.2.7-513.gb88fae66.el9 3.7 MB/s | 320 kB 00:00 2024-02-16T22:09:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:(11/24): python3-rbd-17.2.7-513.gb88fae66.el9.x 3.1 MB/s | 293 kB 00:00 2024-02-16T22:09:02.049 INFO:teuthology.orchestra.run.smithi002.stdout:(12/24): python3-rgw-17.2.7-513.gb88fae66.el9.x 1.1 MB/s | 99 kB 00:00 2024-02-16T22:09:02.116 INFO:teuthology.orchestra.run.smithi002.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-02-16T22:09:02.158 INFO:teuthology.orchestra.run.smithi002.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 412 kB/s | 44 kB 00:00 2024-02-16T22:09:02.200 INFO:teuthology.orchestra.run.smithi002.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-02-16T22:09:02.225 INFO:teuthology.orchestra.run.smithi002.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-02-16T22:09:02.250 INFO:teuthology.orchestra.run.smithi002.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2024-02-16T22:09:02.334 INFO:teuthology.orchestra.run.smithi002.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-02-16T22:09:02.409 INFO:teuthology.orchestra.run.smithi002.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-02-16T22:09:02.434 INFO:teuthology.orchestra.run.smithi002.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2024-02-16T22:09:02.618 INFO:teuthology.orchestra.run.smithi002.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-02-16T22:09:02.711 INFO:teuthology.orchestra.run.smithi103.stdout:(21/24): librados2-17.2.7-513.gb88fae66.el9.x86 2.2 MB/s | 3.1 MB 00:01 2024-02-16T22:09:02.960 INFO:teuthology.orchestra.run.smithi002.stdout:(22/24): librados2-17.2.7-513.gb88fae66.el9.x86 5.9 MB/s | 3.1 MB 00:00 2024-02-16T22:09:03.010 INFO:teuthology.orchestra.run.smithi002.stdout:(23/24): librbd1-17.2.7-513.gb88fae66.el9.x86_6 7.5 MB/s | 2.9 MB 00:00 2024-02-16T22:09:03.444 INFO:teuthology.orchestra.run.smithi103.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 756 kB/s | 1.6 MB 00:02 2024-02-16T22:09:03.494 INFO:teuthology.orchestra.run.smithi103.stdout:(23/24): librbd1-17.2.7-513.gb88fae66.el9.x86_6 3.7 MB/s | 2.9 MB 00:00 2024-02-16T22:09:04.077 INFO:teuthology.orchestra.run.smithi002.stdout:(24/24): ceph-common-17.2.7-513.gb88fae66.el9.x 4.1 MB/s | 23 MB 00:05 2024-02-16T22:09:04.081 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:04.081 INFO:teuthology.orchestra.run.smithi002.stdout:Total 10 MB/s | 58 MB 00:05 2024-02-16T22:09:04.081 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:09:04.118 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:09:04.118 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:09:04.603 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:09:04.604 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:09:05.478 INFO:teuthology.orchestra.run.smithi103.stdout:(24/24): ceph-common-17.2.7-513.gb88fae66.el9.x 2.9 MB/s | 23 MB 00:07 2024-02-16T22:09:05.483 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:05.483 INFO:teuthology.orchestra.run.smithi103.stdout:Total 7.4 MB/s | 58 MB 00:07 2024-02-16T22:09:05.484 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:09:05.516 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:09:05.517 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:09:05.897 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:09:05.999 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:09:05.999 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:09:06.247 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-16T22:09:06.288 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librados2-2:17.2.7-513.gb88fae66.el9.x86_64 2/26 2024-02-16T22:09:06.372 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:17.2.7-513.gb88fae66.el9.x86_64 2/26 2024-02-16T22:09:06.452 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rados-2:17.2.7-513.gb88fae66.el9.x86_64 3/26 2024-02-16T22:09:06.481 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 4/26 2024-02-16T22:09:06.570 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 4/26 2024-02-16T22:09:06.623 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-16T22:09:06.660 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-16T22:09:07.119 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-16T22:09:07.150 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librgw2-2:17.2.7-513.gb88fae66.el9.x86_64 8/26 2024-02-16T22:09:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librgw2-2:17.2.7-513.gb88fae66.el9.x86_64 8/26 2024-02-16T22:09:07.218 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:09:07.242 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-16T22:09:07.285 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-argparse-2:17.2.7-513.gb88fae66.el9.x 10/26 2024-02-16T22:09:07.330 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cephfs-2:17.2.7-513.gb88fae66.el9.x86_64 11/26 2024-02-16T22:09:07.393 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rgw-2:17.2.7-513.gb88fae66.el9.x86_64 12/26 2024-02-16T22:09:07.419 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libradosstriper1-2:17.2.7-513.gb88fae66.el9.x86_64 13/26 2024-02-16T22:09:07.485 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libradosstriper1-2:17.2.7-513.gb88fae66.el9.x86_64 13/26 2024-02-16T22:09:07.528 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-16T22:09:07.557 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-16T22:09:07.561 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-16T22:09:07.594 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-16T22:09:07.604 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:17.2.7-513.gb88fae66.el9.x86_64 2/26 2024-02-16T22:09:07.678 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:17.2.7-513.gb88fae66.el9.x86_64 2/26 2024-02-16T22:09:07.758 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:17.2.7-513.gb88fae66.el9.x86_64 3/26 2024-02-16T22:09:07.789 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 4/26 2024-02-16T22:09:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 4/26 2024-02-16T22:09:07.904 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-16T22:09:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-16T22:09:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 18/26 2024-02-16T22:09:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-16T22:09:08.030 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 18/26 2024-02-16T22:09:08.105 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rbd-2:17.2.7-513.gb88fae66.el9.x86_64 19/26 2024-02-16T22:09:08.134 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-common-2:17.2.7-513.gb88fae66.el9.x86 20/26 2024-02-16T22:09:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-16T22:09:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:17.2.7-513.gb88fae66.el9.x86_64 8/26 2024-02-16T22:09:08.472 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:17.2.7-513.gb88fae66.el9.x86_64 8/26 2024-02-16T22:09:08.500 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-16T22:09:08.543 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:17.2.7-513.gb88fae66.el9.x 10/26 2024-02-16T22:09:08.580 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:17.2.7-513.gb88fae66.el9.x86_64 11/26 2024-02-16T22:09:08.650 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:17.2.7-513.gb88fae66.el9.x86_64 12/26 2024-02-16T22:09:08.678 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:17.2.7-513.gb88fae66.el9.x86_64 13/26 2024-02-16T22:09:08.734 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:17.2.7-513.gb88fae66.el9.x86_64 13/26 2024-02-16T22:09:08.775 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-16T22:09:08.820 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-16T22:09:08.853 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-16T22:09:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-16T22:09:09.166 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 18/26 2024-02-16T22:09:09.258 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 18/26 2024-02-16T22:09:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:17.2.7-513.gb88fae66.el9.x86_64 19/26 2024-02-16T22:09:09.360 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:17.2.7-513.gb88fae66.el9.x86 20/26 2024-02-16T22:09:10.672 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 21/26 2024-02-16T22:09:10.705 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 21/26 2024-02-16T22:09:11.285 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 21/26 2024-02-16T22:09:11.317 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 22/26 2024-02-16T22:09:11.408 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 22/26 2024-02-16T22:09:11.408 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-16T22:09:11.408 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-16T22:09:11.409 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:11.443 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-selinux-2:17.2.7-513.gb88fae66.el9.x86_64 23/26 2024-02-16T22:09:11.846 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 21/26 2024-02-16T22:09:11.881 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 21/26 2024-02-16T22:09:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 21/26 2024-02-16T22:09:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 22/26 2024-02-16T22:09:12.551 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 22/26 2024-02-16T22:09:12.551 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-16T22:09:12.551 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-16T22:09:12.552 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:12.594 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:17.2.7-513.gb88fae66.el9.x86_64 23/26 2024-02-16T22:09:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-selinux-2:17.2.7-513.gb88fae66.el9.x86_64 23/26 2024-02-16T22:09:21.612 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /sys 2024-02-16T22:09:21.612 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /proc 2024-02-16T22:09:21.613 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /mnt 2024-02-16T22:09:21.613 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /var/tmp 2024-02-16T22:09:21.613 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /home 2024-02-16T22:09:21.613 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /root 2024-02-16T22:09:21.613 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /tmp 2024-02-16T22:09:21.613 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:17.2.7-513.gb88fae66.el9.x86_64 23/26 2024-02-16T22:09:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-02-16T22:09:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-02-16T22:09:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-02-16T22:09:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-02-16T22:09:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-02-16T22:09:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-02-16T22:09:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-02-16T22:09:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:22.944 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-radosgw-2:17.2.7-513.gb88fae66.el9.x86_64 24/26 2024-02-16T22:09:23.004 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-513.gb88fae66.el9.x86_64 24/26 2024-02-16T22:09:23.005 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:23.005 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-16T22:09:23.005 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-16T22:09:23.005 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-16T22:09:23.005 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:23.006 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-16T22:09:23.071 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-16T22:09:23.072 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-16T22:09:23.411 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:17.2.7-513.gb88fae66.el9.x86_64 24/26 2024-02-16T22:09:23.473 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-513.gb88fae66.el9.x86_64 24/26 2024-02-16T22:09:23.473 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:23.473 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-16T22:09:23.473 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-16T22:09:23.473 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-16T22:09:23.473 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:23.475 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-16T22:09:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-16T22:09:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-16T22:09:24.320 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 1/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 2/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-radosgw-2:17.2.7-513.gb88fae66.el9.x86_64 3/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-selinux-2:17.2.7-513.gb88fae66.el9.x86_64 4/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 5/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libradosstriper1-2:17.2.7-513.gb88fae66.el9.x86_64 6/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librgw2-2:17.2.7-513.gb88fae66.el9.x86_64 7/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-argparse-2:17.2.7-513.gb88fae66.el9.x 8/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-common-2:17.2.7-513.gb88fae66.el9.x86 9/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cephfs-2:17.2.7-513.gb88fae66.el9.x86_64 10/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rados-2:17.2.7-513.gb88fae66.el9.x86_64 11/26 2024-02-16T22:09:24.321 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rbd-2:17.2.7-513.gb88fae66.el9.x86_64 12/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rgw-2:17.2.7-513.gb88fae66.el9.x86_64 13/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:17.2.7-513.gb88fae66.el9.x86_64 23/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-16T22:09:24.322 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 25/26 2024-02-16T22:09:24.807 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-16T22:09:24.807 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 1/26 2024-02-16T22:09:24.807 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 2/26 2024-02-16T22:09:24.807 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:17.2.7-513.gb88fae66.el9.x86_64 3/26 2024-02-16T22:09:24.807 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:17.2.7-513.gb88fae66.el9.x86_64 4/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 5/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:17.2.7-513.gb88fae66.el9.x86_64 6/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:17.2.7-513.gb88fae66.el9.x86_64 7/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:17.2.7-513.gb88fae66.el9.x 8/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:17.2.7-513.gb88fae66.el9.x86 9/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:17.2.7-513.gb88fae66.el9.x86_64 10/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:17.2.7-513.gb88fae66.el9.x86_64 11/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:17.2.7-513.gb88fae66.el9.x86_64 12/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:17.2.7-513.gb88fae66.el9.x86_64 13/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-16T22:09:24.808 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-16T22:09:24.809 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-16T22:09:24.809 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-16T22:09:24.809 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-16T22:09:24.809 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-16T22:09:24.809 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-16T22:09:24.809 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:17.2.7-513.gb88fae66.el9.x86_64 23/26 2024-02-16T22:09:24.809 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-16T22:09:24.809 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 25/26 2024-02-16T22:09:25.225 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: librados2-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-16T22:09:25.226 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.227 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.228 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.228 INFO:teuthology.orchestra.run.smithi002.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-16T22:09:25.228 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:25.228 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:09:25.542 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-test 2024-02-16T22:09:25.706 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.707 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-16T22:09:25.708 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.709 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:25.709 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-16T22:09:25.709 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:25.709 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:09:26.073 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-test 2024-02-16T22:09:26.127 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:31 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:26.303 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:09:26.304 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:09:26.304 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:09:26.304 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:09:26.304 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:09:26.304 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test x86_64 2:17.2.7-513.gb88fae66.el9 ceph 48 M 2024-02-16T22:09:26.304 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite x86_64 2:17.2.7-513.gb88fae66.el9 ceph 162 k 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout:Install 6 Packages 2024-02-16T22:09:26.305 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:26.306 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 49 M 2024-02-16T22:09:26.306 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 196 M 2024-02-16T22:09:26.306 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:09:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 300 kB 00:00 2024-02-16T22:09:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-16T22:09:26.566 INFO:teuthology.orchestra.run.smithi002.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.3 MB/s | 186 kB 00:00 2024-02-16T22:09:26.591 INFO:teuthology.orchestra.run.smithi002.stdout:(4/6): libcephsqlite-17.2.7-513.gb88fae66.el9.x 570 kB/s | 162 kB 00:00 2024-02-16T22:09:26.617 INFO:teuthology.orchestra.run.smithi002.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-02-16T22:09:26.643 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:31 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:26.811 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:09:26.812 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:09:26.812 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:09:26.812 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:09:26.812 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:09:26.813 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:17.2.7-513.gb88fae66.el9 ceph 48 M 2024-02-16T22:09:26.813 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-16T22:09:26.813 INFO:teuthology.orchestra.run.smithi103.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-16T22:09:26.813 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:17.2.7-513.gb88fae66.el9 ceph 162 k 2024-02-16T22:09:26.813 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-16T22:09:26.814 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-16T22:09:26.814 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-16T22:09:26.814 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:26.814 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:09:26.814 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:09:26.814 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-02-16T22:09:26.814 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:26.815 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 49 M 2024-02-16T22:09:26.815 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 196 M 2024-02-16T22:09:26.816 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:09:26.981 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2024-02-16T22:09:27.006 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-02-16T22:09:27.048 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.3 MB/s | 186 kB 00:00 2024-02-16T22:09:27.097 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.4 MB/s | 217 kB 00:00 2024-02-16T22:09:27.198 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): libcephsqlite-17.2.7-513.gb88fae66.el9.x 423 kB/s | 162 kB 00:00 2024-02-16T22:09:29.142 INFO:teuthology.orchestra.run.smithi002.stdout:(6/6): ceph-test-17.2.7-513.gb88fae66.el9.x86_6 17 MB/s | 48 MB 00:02 2024-02-16T22:09:29.145 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:29.146 INFO:teuthology.orchestra.run.smithi002.stdout:Total 17 MB/s | 49 MB 00:02 2024-02-16T22:09:29.146 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:09:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:09:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:09:29.571 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:09:29.572 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:09:30.170 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:09:30.251 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-16T22:09:30.292 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-16T22:09:30.336 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-16T22:09:30.377 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-16T22:09:30.403 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephsqlite-2:17.2.7-513.gb88fae66.el9.x86_64 5/6 2024-02-16T22:09:31.098 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): ceph-test-17.2.7-513.gb88fae66.el9.x86_6 11 MB/s | 48 MB 00:04 2024-02-16T22:09:31.101 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:31.102 INFO:teuthology.orchestra.run.smithi103.stdout:Total 11 MB/s | 49 MB 00:04 2024-02-16T22:09:31.102 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:09:31.121 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:09:31.122 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:09:31.509 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:09:31.509 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:09:32.111 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:09:32.174 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-16T22:09:32.210 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-16T22:09:32.256 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-16T22:09:32.296 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-16T22:09:32.323 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:17.2.7-513.gb88fae66.el9.x86_64 5/6 2024-02-16T22:09:35.069 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephsqlite-2:17.2.7-513.gb88fae66.el9.x86_64 5/6 2024-02-16T22:09:35.154 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-test-2:17.2.7-513.gb88fae66.el9.x86_64 6/6 2024-02-16T22:09:36.733 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:17.2.7-513.gb88fae66.el9.x86_64 5/6 2024-02-16T22:09:36.814 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:17.2.7-513.gb88fae66.el9.x86_64 6/6 2024-02-16T22:09:37.343 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-test-2:17.2.7-513.gb88fae66.el9.x86_64 6/6 2024-02-16T22:09:37.343 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-test-2:17.2.7-513.gb88fae66.el9.x86_64 1/6 2024-02-16T22:09:37.343 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephsqlite-2:17.2.7-513.gb88fae66.el9.x86_64 2/6 2024-02-16T22:09:37.343 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-16T22:09:37.343 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-16T22:09:37.343 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-16T22:09:37.761 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-16T22:09:37.761 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:37.761 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:09:37.761 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:37.761 INFO:teuthology.orchestra.run.smithi002.stdout: jq-1.6-16.el9.x86_64 2024-02-16T22:09:37.762 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:37.762 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-16T22:09:37.762 INFO:teuthology.orchestra.run.smithi002.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-16T22:09:37.762 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-16T22:09:37.762 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:37.762 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:09:38.064 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph 2024-02-16T22:09:38.660 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:43 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:38.847 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:09:38.849 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================== 2024-02-16T22:09:38.849 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:09:38.849 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================== 2024-02-16T22:09:38.849 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:09:38.849 INFO:teuthology.orchestra.run.smithi002.stdout: ceph x86_64 2:17.2.7-513.gb88fae66.el9 ceph 6.5 k 2024-02-16T22:09:38.849 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-16T22:09:38.849 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds x86_64 2:17.2.7-513.gb88fae66.el9 ceph 2.1 M 2024-02-16T22:09:38.849 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr x86_64 2:17.2.7-513.gb88fae66.el9 ceph 1.4 M 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 239 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon x86_64 2:17.2.7-513.gb88fae66.el9 ceph 3.7 M 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd x86_64 2:17.2.7-513.gb88fae66.el9 ceph 15 M 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-16T22:09:38.850 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-16T22:09:38.851 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-16T22:09:38.852 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:38.852 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:09:38.852 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================== 2024-02-16T22:09:38.852 INFO:teuthology.orchestra.run.smithi002.stdout:Install 33 Packages 2024-02-16T22:09:38.852 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:38.852 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 27 M 2024-02-16T22:09:38.853 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 99 M 2024-02-16T22:09:38.853 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:09:39.211 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-test-2:17.2.7-513.gb88fae66.el9.x86_64 6/6 2024-02-16T22:09:39.211 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:17.2.7-513.gb88fae66.el9.x86_64 1/6 2024-02-16T22:09:39.211 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:17.2.7-513.gb88fae66.el9.x86_64 2/6 2024-02-16T22:09:39.211 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-16T22:09:39.211 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-16T22:09:39.211 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-16T22:09:39.535 INFO:teuthology.orchestra.run.smithi002.stdout:(1/33): ceph-17.2.7-513.gb88fae66.el9.x86_64.rp 20 kB/s | 6.5 kB 00:00 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout: jq-1.6-16.el9.x86_64 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:39.579 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:09:39.891 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph 2024-02-16T22:09:40.499 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:45 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:40.518 INFO:teuthology.orchestra.run.smithi002.stdout:(2/33): ceph-mds-17.2.7-513.gb88fae66.el9.x86_6 1.6 MB/s | 2.1 MB 00:01 2024-02-16T22:09:40.681 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:17.2.7-513.gb88fae66.el9 ceph 6.5 k 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:17.2.7-513.gb88fae66.el9 ceph 2.1 M 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:17.2.7-513.gb88fae66.el9 ceph 1.4 M 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 239 k 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:17.2.7-513.gb88fae66.el9 ceph 3.7 M 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:17.2.7-513.gb88fae66.el9 ceph 15 M 2024-02-16T22:09:40.684 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-16T22:09:40.685 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================== 2024-02-16T22:09:40.686 INFO:teuthology.orchestra.run.smithi103.stdout:Install 33 Packages 2024-02-16T22:09:40.687 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:40.687 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 27 M 2024-02-16T22:09:40.687 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 99 M 2024-02-16T22:09:40.687 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:09:40.752 INFO:teuthology.orchestra.run.smithi002.stdout:(3/33): ceph-mgr-17.2.7-513.gb88fae66.el9.x86_6 930 kB/s | 1.4 MB 00:01 2024-02-16T22:09:40.986 INFO:teuthology.orchestra.run.smithi002.stdout:(4/33): ceph-mgr-modules-core-17.2.7-513.gb88fa 1.0 MB/s | 239 kB 00:00 2024-02-16T22:09:41.261 INFO:teuthology.orchestra.run.smithi002.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 524 kB/s | 144 kB 00:00 2024-02-16T22:09:41.315 INFO:teuthology.orchestra.run.smithi103.stdout:(1/33): ceph-mgr-17.2.7-513.gb88fae66.el9.x86_6 4.0 MB/s | 1.4 MB 00:00 2024-02-16T22:09:41.328 INFO:teuthology.orchestra.run.smithi002.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 474 kB/s | 32 kB 00:00 2024-02-16T22:09:41.340 INFO:teuthology.orchestra.run.smithi103.stdout:(2/33): ceph-17.2.7-513.gb88fae66.el9.x86_64.rp 17 kB/s | 6.5 kB 00:00 2024-02-16T22:09:41.403 INFO:teuthology.orchestra.run.smithi002.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 633 kB/s | 47 kB 00:00 2024-02-16T22:09:41.407 INFO:teuthology.orchestra.run.smithi103.stdout:(3/33): ceph-mds-17.2.7-513.gb88fae66.el9.x86_6 4.7 MB/s | 2.1 MB 00:00 2024-02-16T22:09:41.470 INFO:teuthology.orchestra.run.smithi002.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 579 kB/s | 39 kB 00:00 2024-02-16T22:09:41.499 INFO:teuthology.orchestra.run.smithi103.stdout:(4/33): ceph-mgr-modules-core-17.2.7-513.gb88fa 2.5 MB/s | 239 kB 00:00 2024-02-16T22:09:41.629 INFO:teuthology.orchestra.run.smithi002.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 241 kB 00:00 2024-02-16T22:09:41.841 INFO:teuthology.orchestra.run.smithi103.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 422 kB/s | 144 kB 00:00 2024-02-16T22:09:41.854 INFO:teuthology.orchestra.run.smithi002.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 5.1 MB/s | 1.1 MB 00:00 2024-02-16T22:09:41.908 INFO:teuthology.orchestra.run.smithi103.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 474 kB/s | 32 kB 00:00 2024-02-16T22:09:41.921 INFO:teuthology.orchestra.run.smithi002.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-02-16T22:09:41.983 INFO:teuthology.orchestra.run.smithi103.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-02-16T22:09:41.996 INFO:teuthology.orchestra.run.smithi002.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 124 kB 00:00 2024-02-16T22:09:42.088 INFO:teuthology.orchestra.run.smithi002.stdout:(13/33): python3-requests-2.25.1-8.el9.noarch.r 1.2 MB/s | 113 kB 00:00 2024-02-16T22:09:42.163 INFO:teuthology.orchestra.run.smithi002.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-02-16T22:09:42.175 INFO:teuthology.orchestra.run.smithi103.stdout:(8/33): ceph-osd-17.2.7-513.gb88fae66.el9.x86_6 18 MB/s | 15 MB 00:00 2024-02-16T22:09:42.200 INFO:teuthology.orchestra.run.smithi103.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 178 kB/s | 39 kB 00:00 2024-02-16T22:09:42.326 INFO:teuthology.orchestra.run.smithi103.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 241 kB 00:00 2024-02-16T22:09:42.526 INFO:teuthology.orchestra.run.smithi103.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 3.5 MB/s | 1.1 MB 00:00 2024-02-16T22:09:42.547 INFO:teuthology.orchestra.run.smithi002.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 113 kB/s | 43 kB 00:00 2024-02-16T22:09:42.551 INFO:teuthology.orchestra.run.smithi103.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 457 kB/s | 103 kB 00:00 2024-02-16T22:09:42.685 INFO:teuthology.orchestra.run.smithi103.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 930 kB/s | 124 kB 00:00 2024-02-16T22:09:42.710 INFO:teuthology.orchestra.run.smithi103.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 713 kB/s | 113 kB 00:00 2024-02-16T22:09:42.877 INFO:teuthology.orchestra.run.smithi103.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 977 kB/s | 187 kB 00:00 2024-02-16T22:09:43.103 INFO:teuthology.orchestra.run.smithi103.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 111 kB/s | 43 kB 00:00 2024-02-16T22:09:43.147 INFO:teuthology.orchestra.run.smithi002.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 286 kB/s | 172 kB 00:00 2024-02-16T22:09:43.336 INFO:teuthology.orchestra.run.smithi103.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 374 kB/s | 172 kB 00:00 2024-02-16T22:09:43.428 INFO:teuthology.orchestra.run.smithi103.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-02-16T22:09:43.478 INFO:teuthology.orchestra.run.smithi103.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 954 kB/s | 358 kB 00:00 2024-02-16T22:09:43.520 INFO:teuthology.orchestra.run.smithi103.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-02-16T22:09:43.571 INFO:teuthology.orchestra.run.smithi103.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-02-16T22:09:43.612 INFO:teuthology.orchestra.run.smithi103.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-02-16T22:09:43.656 INFO:teuthology.orchestra.run.smithi002.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 705 kB/s | 358 kB 00:00 2024-02-16T22:09:43.663 INFO:teuthology.orchestra.run.smithi103.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-02-16T22:09:43.689 INFO:teuthology.orchestra.run.smithi002.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 322 kB/s | 11 kB 00:00 2024-02-16T22:09:43.722 INFO:teuthology.orchestra.run.smithi103.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 425 kB/s | 46 kB 00:00 2024-02-16T22:09:43.723 INFO:teuthology.orchestra.run.smithi002.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 531 kB/s | 18 kB 00:00 2024-02-16T22:09:43.757 INFO:teuthology.orchestra.run.smithi002.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 695 kB/s | 23 kB 00:00 2024-02-16T22:09:43.772 INFO:teuthology.orchestra.run.smithi103.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-02-16T22:09:43.807 INFO:teuthology.orchestra.run.smithi002.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 391 kB/s | 19 kB 00:00 2024-02-16T22:09:43.849 INFO:teuthology.orchestra.run.smithi002.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 475 kB/s | 20 kB 00:00 2024-02-16T22:09:43.873 INFO:teuthology.orchestra.run.smithi103.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 163 kB/s | 16 kB 00:00 2024-02-16T22:09:43.898 INFO:teuthology.orchestra.run.smithi103.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-02-16T22:09:43.916 INFO:teuthology.orchestra.run.smithi002.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 695 kB/s | 46 kB 00:00 2024-02-16T22:09:43.981 INFO:teuthology.orchestra.run.smithi103.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 829 kB/s | 90 kB 00:00 2024-02-16T22:09:44.007 INFO:teuthology.orchestra.run.smithi103.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-02-16T22:09:44.116 INFO:teuthology.orchestra.run.smithi103.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-02-16T22:09:44.133 INFO:teuthology.orchestra.run.smithi002.stdout:(24/33): ceph-osd-17.2.7-513.gb88fae66.el9.x86_ 4.2 MB/s | 15 MB 00:03 2024-02-16T22:09:44.149 INFO:teuthology.orchestra.run.smithi103.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-02-16T22:09:44.158 INFO:teuthology.orchestra.run.smithi002.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 326 kB/s | 79 kB 00:00 2024-02-16T22:09:44.216 INFO:teuthology.orchestra.run.smithi103.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-02-16T22:09:44.417 INFO:teuthology.orchestra.run.smithi002.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 961 kB/s | 272 kB 00:00 2024-02-16T22:09:44.442 INFO:teuthology.orchestra.run.smithi002.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 58 kB/s | 16 kB 00:00 2024-02-16T22:09:44.542 INFO:teuthology.orchestra.run.smithi002.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 719 kB/s | 90 kB 00:00 2024-02-16T22:09:44.576 INFO:teuthology.orchestra.run.smithi002.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 268 kB/s | 36 kB 00:00 2024-02-16T22:09:44.793 INFO:teuthology.orchestra.run.smithi002.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 918 kB/s | 230 kB 00:00 2024-02-16T22:09:44.860 INFO:teuthology.orchestra.run.smithi002.stdout:(31/33): ceph-mon-17.2.7-513.gb88fae66.el9.x86_ 711 kB/s | 3.7 MB 00:05 2024-02-16T22:09:45.143 INFO:teuthology.orchestra.run.smithi002.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 753 kB/s | 427 kB 00:00 2024-02-16T22:09:45.168 INFO:teuthology.orchestra.run.smithi002.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 53 kB/s | 20 kB 00:00 2024-02-16T22:09:45.173 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:45.173 INFO:teuthology.orchestra.run.smithi002.stdout:Total 4.2 MB/s | 27 MB 00:06 2024-02-16T22:09:45.173 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:09:45.216 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:09:45.216 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:09:45.570 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:09:45.571 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:09:45.708 INFO:teuthology.orchestra.run.smithi103.stdout:(33/33): ceph-mon-17.2.7-513.gb88fae66.el9.x86_ 861 kB/s | 3.7 MB 00:04 2024-02-16T22:09:45.711 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:45.711 INFO:teuthology.orchestra.run.smithi103.stdout:Total 5.3 MB/s | 27 MB 00:05 2024-02-16T22:09:45.712 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:09:45.751 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:09:45.751 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:09:46.088 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:09:46.088 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:09:46.810 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:09:46.880 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-16T22:09:46.912 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-16T22:09:46.945 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-16T22:09:47.039 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-16T22:09:47.116 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-16T22:09:47.175 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-16T22:09:47.254 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-16T22:09:47.304 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:09:47.307 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-16T22:09:47.345 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-16T22:09:47.361 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-16T22:09:47.393 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-16T22:09:47.396 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-16T22:09:47.426 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-16T22:09:47.452 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-16T22:09:47.517 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-16T22:09:47.596 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-16T22:09:47.637 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-16T22:09:47.642 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-16T22:09:47.693 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-16T22:09:47.718 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-16T22:09:47.740 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-16T22:09:47.771 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-16T22:09:47.774 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-16T22:09:47.804 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-16T22:09:47.809 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-16T22:09:47.859 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-16T22:09:47.882 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-16T22:09:47.913 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-16T22:09:47.931 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-16T22:09:47.964 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-16T22:09:48.004 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-16T22:09:48.040 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-16T22:09:48.083 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-16T22:09:48.099 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-16T22:09:48.148 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-16T22:09:48.175 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-16T22:09:48.186 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-16T22:09:48.220 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-16T22:09:48.231 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-16T22:09:48.251 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-16T22:09:48.305 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-16T22:09:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-16T22:09:48.362 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-16T22:09:48.395 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-16T22:09:48.427 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-16T22:09:48.443 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-16T22:09:48.454 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-16T22:09:48.488 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-16T22:09:48.567 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-16T22:09:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-16T22:09:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-16T22:09:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-16T22:09:48.734 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-513.gb88fae66.el9.n 28/33 2024-02-16T22:09:48.759 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 29/33 2024-02-16T22:09:48.785 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 29/33 2024-02-16T22:09:48.785 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:48.785 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-16T22:09:48.785 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-16T22:09:48.785 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-16T22:09:48.785 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:48.838 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-16T22:09:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-16T22:09:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-513.gb88fae66.el9.n 28/33 2024-02-16T22:09:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 29/33 2024-02-16T22:09:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 29/33 2024-02-16T22:09:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-16T22:09:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-16T22:09:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-16T22:09:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:50.211 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-osd-2:17.2.7-513.gb88fae66.el9.x86_64 30/33 2024-02-16T22:09:50.237 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-osd-2:17.2.7-513.gb88fae66.el9.x86_64 30/33 2024-02-16T22:09:50.237 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:50.237 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-16T22:09:50.237 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-16T22:09:50.237 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-16T22:09:50.237 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:50.517 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:17.2.7-513.gb88fae66.el9.x86_64 30/33 2024-02-16T22:09:50.544 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:17.2.7-513.gb88fae66.el9.x86_64 30/33 2024-02-16T22:09:50.544 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:50.544 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-16T22:09:50.544 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-16T22:09:50.544 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-16T22:09:50.544 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:50.628 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mon-2:17.2.7-513.gb88fae66.el9.x86_64 31/33 2024-02-16T22:09:50.650 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mon-2:17.2.7-513.gb88fae66.el9.x86_64 31/33 2024-02-16T22:09:50.650 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:50.651 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-16T22:09:50.651 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-16T22:09:50.651 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-16T22:09:50.651 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:50.862 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mds-2:17.2.7-513.gb88fae66.el9.x86_64 32/33 2024-02-16T22:09:50.885 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mds-2:17.2.7-513.gb88fae66.el9.x86_64 32/33 2024-02-16T22:09:50.885 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:50.885 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-16T22:09:50.885 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-16T22:09:50.885 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-16T22:09:50.885 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:17.2.7-513.gb88fae66.el9.x86_64 31/33 2024-02-16T22:09:50.949 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:17.2.7-513.gb88fae66.el9.x86_64 31/33 2024-02-16T22:09:50.949 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:50.950 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-16T22:09:50.950 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-16T22:09:50.950 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-16T22:09:50.950 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:50.953 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-2:17.2.7-513.gb88fae66.el9.x86_64 33/33 2024-02-16T22:09:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:17.2.7-513.gb88fae66.el9.x86_64 32/33 2024-02-16T22:09:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:17.2.7-513.gb88fae66.el9.x86_64 32/33 2024-02-16T22:09:51.184 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:51.184 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-16T22:09:51.184 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-16T22:09:51.184 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-16T22:09:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:51.252 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:17.2.7-513.gb88fae66.el9.x86_64 33/33 2024-02-16T22:09:52.704 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-2:17.2.7-513.gb88fae66.el9.x86_64 33/33 2024-02-16T22:09:52.704 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-2:17.2.7-513.gb88fae66.el9.x86_64 1/33 2024-02-16T22:09:52.704 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mds-2:17.2.7-513.gb88fae66.el9.x86_64 2/33 2024-02-16T22:09:52.704 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 3/33 2024-02-16T22:09:52.704 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mon-2:17.2.7-513.gb88fae66.el9.x86_64 4/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-osd-2:17.2.7-513.gb88fae66.el9.x86_64 5/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-513.gb88fae66.el9.n 6/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-16T22:09:52.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-16T22:09:52.706 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-16T22:09:52.706 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-16T22:09:52.706 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-16T22:09:52.706 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-16T22:09:52.706 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-16T22:09:52.706 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-16T22:09:52.706 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-16T22:09:52.707 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-16T22:09:52.707 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-16T22:09:52.707 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-16T22:09:52.707 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-16T22:09:52.707 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-16T22:09:52.707 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-16T22:09:52.707 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-16T22:09:52.982 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-2:17.2.7-513.gb88fae66.el9.x86_64 33/33 2024-02-16T22:09:52.982 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:17.2.7-513.gb88fae66.el9.x86_64 1/33 2024-02-16T22:09:52.982 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:17.2.7-513.gb88fae66.el9.x86_64 2/33 2024-02-16T22:09:52.982 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 3/33 2024-02-16T22:09:52.982 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:17.2.7-513.gb88fae66.el9.x86_64 4/33 2024-02-16T22:09:52.982 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:17.2.7-513.gb88fae66.el9.x86_64 5/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-513.gb88fae66.el9.n 6/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-16T22:09:52.983 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-16T22:09:52.984 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-16T22:09:52.984 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-16T22:09:52.984 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-16T22:09:52.984 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-16T22:09:52.984 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-16T22:09:52.984 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-16T22:09:52.984 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-16T22:09:52.984 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-16T22:09:52.985 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-16T22:09:52.985 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-16T22:09:52.985 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-16T22:09:52.985 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-16T22:09:53.659 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-16T22:09:53.659 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:53.659 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:09:53.659 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:53.659 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:53.659 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:53.659 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:09:53.659 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-16T22:09:53.660 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ply-3.11-14.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-16T22:09:53.661 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-16T22:09:53.662 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:53.662 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:09:53.929 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-base 2024-02-16T22:09:54.087 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-16T22:09:54.087 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:54.087 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:09:54.087 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-16T22:09:54.088 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-16T22:09:54.089 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-16T22:09:54.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-16T22:09:54.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-16T22:09:54.090 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-16T22:09:54.090 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:54.090 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:09:54.403 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-base 2024-02-16T22:09:54.519 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:59 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:54.637 INFO:teuthology.orchestra.run.smithi002.stdout:Package ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:09:54.691 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:09:54.692 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-02-16T22:09:54.692 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:09:54.752 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install cephadm 2024-02-16T22:09:54.985 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:00:59 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:55.100 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-base-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:09:55.151 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:09:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-02-16T22:09:55.152 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:09:55.222 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install cephadm 2024-02-16T22:09:55.337 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:00 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:55.507 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:09:55.508 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:09:55.508 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:09:55.508 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:09:55.508 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:09:55.508 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 80 k 2024-02-16T22:09:55.508 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:55.508 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:09:55.508 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:09:55.509 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-02-16T22:09:55.509 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:55.509 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 80 k 2024-02-16T22:09:55.509 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 364 k 2024-02-16T22:09:55.509 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:09:55.720 INFO:teuthology.orchestra.run.smithi002.stdout:cephadm-17.2.7-513.gb88fae66.el9.noarch.rpm 381 kB/s | 80 kB 00:00 2024-02-16T22:09:55.721 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:55.721 INFO:teuthology.orchestra.run.smithi002.stdout:Total 378 kB/s | 80 kB 00:00 2024-02-16T22:09:55.721 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:09:55.726 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:09:55.727 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:09:55.728 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:09:55.729 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:09:55.797 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:09:55.798 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:00 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:55.960 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 80 k 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-02-16T22:09:55.961 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 80 k 2024-02-16T22:09:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 364 k 2024-02-16T22:09:55.962 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:09:56.185 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/1 2024-02-16T22:09:56.286 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/1 2024-02-16T22:09:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm-17.2.7-513.gb88fae66.el9.noarch.rpm 179 kB/s | 80 kB 00:00 2024-02-16T22:09:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:56.411 INFO:teuthology.orchestra.run.smithi103.stdout:Total 179 kB/s | 80 kB 00:00 2024-02-16T22:09:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:09:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:09:56.417 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:09:56.418 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:09:56.419 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:09:56.489 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:09:56.728 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/1 2024-02-16T22:09:56.960 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/1 2024-02-16T22:09:57.050 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/1 2024-02-16T22:09:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/1 2024-02-16T22:09:57.061 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:57.062 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:09:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:09:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:57.062 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:09:57.241 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-immutable-object-cache 2024-02-16T22:09:57.410 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/1 2024-02-16T22:09:57.728 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/1 2024-02-16T22:09:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:57.729 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:09:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:09:57.729 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:57.729 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:09:57.831 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:02 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:57.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-immutable-object-cache 2024-02-16T22:09:58.002 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-513.gb88fae66.el9 ceph 137 k 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-02-16T22:09:58.003 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:58.004 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 137 k 2024-02-16T22:09:58.004 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 379 k 2024-02-16T22:09:58.004 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:09:58.260 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-immutable-object-cache-17.2.7-513.gb88fae6 535 kB/s | 137 kB 00:00 2024-02-16T22:09:58.261 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:58.261 INFO:teuthology.orchestra.run.smithi002.stdout:Total 532 kB/s | 137 kB 00:00 2024-02-16T22:09:58.261 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:09:58.269 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:09:58.269 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:09:58.323 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:09:58.323 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:09:58.468 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:09:58.524 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:03 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:09:58.535 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-513.gb88fae66.e 1/1 2024-02-16T22:09:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-513.gb88fae66.e 1/1 2024-02-16T22:09:58.559 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:58.559 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-16T22:09:58.559 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:58.693 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:09:58.693 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:09:58.693 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-513.gb88fae66.el9 ceph 137 k 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 137 k 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 379 k 2024-02-16T22:09:58.694 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:09:58.943 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-immutable-object-cache-17.2.7-513.gb88fae6 551 kB/s | 137 kB 00:00 2024-02-16T22:09:58.944 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:09:58.944 INFO:teuthology.orchestra.run.smithi103.stdout:Total 548 kB/s | 137 kB 00:00 2024-02-16T22:09:58.944 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:09:58.952 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:09:58.952 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:09:59.003 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:09:59.003 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:09:59.139 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:09:59.209 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-513.gb88fae66.e 1/1 2024-02-16T22:09:59.233 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-513.gb88fae66.e 1/1 2024-02-16T22:09:59.234 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:09:59.234 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-16T22:09:59.234 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:09:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-513.gb88fae66.e 1/1 2024-02-16T22:09:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:59.552 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:09:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:09:59.552 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:09:59.552 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:09:59.788 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr 2024-02-16T22:10:00.209 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-513.gb88fae66.e 1/1 2024-02-16T22:10:00.209 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:00.209 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:00.209 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:00.209 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:00.209 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:00.379 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:05 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:00.432 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr 2024-02-16T22:10:00.497 INFO:teuthology.orchestra.run.smithi002.stdout:Package ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:00.551 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:00.553 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-02-16T22:10:00.553 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:00.611 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-dashboard 2024-02-16T22:10:01.021 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:06 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:01.133 INFO:teuthology.orchestra.run.smithi103.stdout:Package ceph-mgr-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:01.184 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:01.185 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-02-16T22:10:01.185 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:01.191 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:06 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:01.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-dashboard 2024-02-16T22:10:01.365 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:01.366 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:10:01.366 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:01.366 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 1.7 M 2024-02-16T22:10:01.366 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-16T22:10:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 23 k 2024-02-16T22:10:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 15 k 2024-02-16T22:10:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-16T22:10:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-16T22:10:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-16T22:10:01.367 INFO:teuthology.orchestra.run.smithi002.stdout:Installing weak dependencies: 2024-02-16T22:10:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-16T22:10:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:01.367 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:01.367 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:01.367 INFO:teuthology.orchestra.run.smithi002.stdout:Install 7 Packages 2024-02-16T22:10:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:01.368 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 2.0 M 2024-02-16T22:10:01.368 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 8.1 M 2024-02-16T22:10:01.368 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:01.817 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:06 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:01.982 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:01.983 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:01.983 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:10:01.983 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 1.7 M 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 23 k 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 15 k 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-02-16T22:10:01.984 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-16T22:10:01.985 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:01.985 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:01.985 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:01.985 INFO:teuthology.orchestra.run.smithi103.stdout:Install 7 Packages 2024-02-16T22:10:01.985 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:01.985 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.0 M 2024-02-16T22:10:01.985 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 8.1 M 2024-02-16T22:10:01.985 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:02.088 INFO:teuthology.orchestra.run.smithi002.stdout:(1/7): ceph-grafana-dashboards-17.2.7-513.gb88f 62 kB/s | 23 kB 00:00 2024-02-16T22:10:02.147 INFO:teuthology.orchestra.run.smithi002.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 155 kB/s | 9.0 kB 00:00 2024-02-16T22:10:02.213 INFO:teuthology.orchestra.run.smithi103.stdout:(1/7): ceph-prometheus-alerts-17.2.7-513.gb88fa 79 kB/s | 15 kB 00:00 2024-02-16T22:10:02.214 INFO:teuthology.orchestra.run.smithi002.stdout:(3/7): python3-jwt-2.4.0-1.el9.noarch.rpm 614 kB/s | 41 kB 00:00 2024-02-16T22:10:02.264 INFO:teuthology.orchestra.run.smithi002.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 614 kB/s | 31 kB 00:00 2024-02-16T22:10:02.338 INFO:teuthology.orchestra.run.smithi103.stdout:(2/7): ceph-grafana-dashboards-17.2.7-513.gb88f 75 kB/s | 23 kB 00:00 2024-02-16T22:10:02.347 INFO:teuthology.orchestra.run.smithi002.stdout:(5/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-02-16T22:10:02.355 INFO:teuthology.orchestra.run.smithi103.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 64 kB/s | 9.0 kB 00:00 2024-02-16T22:10:02.481 INFO:teuthology.orchestra.run.smithi103.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 288 kB/s | 41 kB 00:00 2024-02-16T22:10:02.498 INFO:teuthology.orchestra.run.smithi103.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 259 kB/s | 31 kB 00:00 2024-02-16T22:10:02.631 INFO:teuthology.orchestra.run.smithi103.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-02-16T22:10:03.639 INFO:teuthology.orchestra.run.smithi002.stdout:(6/7): ceph-mgr-dashboard-17.2.7-513.gb88fae66. 900 kB/s | 1.7 MB 00:01 2024-02-16T22:10:03.981 INFO:teuthology.orchestra.run.smithi103.stdout:(7/7): ceph-mgr-dashboard-17.2.7-513.gb88fae66. 889 kB/s | 1.7 MB 00:01 2024-02-16T22:10:03.985 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:03.985 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.0 MB/s | 2.0 MB 00:02 2024-02-16T22:10:03.985 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:03.994 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:03.994 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:04.062 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:04.063 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:04.156 INFO:teuthology.orchestra.run.smithi002.stdout:(7/7): ceph-prometheus-alerts-17.2.7-513.gb88fa 6.0 kB/s | 15 kB 00:02 2024-02-16T22:10:04.159 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:04.159 INFO:teuthology.orchestra.run.smithi002.stdout:Total 732 kB/s | 2.0 MB 00:02 2024-02-16T22:10:04.159 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:04.169 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:04.169 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:04.247 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:04.247 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-16T22:10:04.467 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-16T22:10:04.513 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-16T22:10:04.545 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-16T22:10:04.567 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:04.574 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-513.gb88fae66.el9.no 5/7 2024-02-16T22:10:04.624 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-16T22:10:04.658 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-16T22:10:04.705 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-16T22:10:04.736 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-16T22:10:04.765 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-513.gb88fae66.el9.no 5/7 2024-02-16T22:10:04.802 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-513.gb88fae66.el9.n 6/7 2024-02-16T22:10:04.836 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-513.gb88fae66.el9.noarch 7/7 2024-02-16T22:10:05.005 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-513.gb88fae66.el9.n 6/7 2024-02-16T22:10:05.043 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-513.gb88fae66.el9.noarch 7/7 2024-02-16T22:10:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-513.gb88fae66.el9.noarch 7/7 2024-02-16T22:10:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-513.gb88fae66.el9.n 1/7 2024-02-16T22:10:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-513.gb88fae66.el9.noarch 2/7 2024-02-16T22:10:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-513.gb88fae66.el9.no 3/7 2024-02-16T22:10:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-16T22:10:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-16T22:10:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-16T22:10:05.590 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-513.gb88fae66.el9.noarch 7/7 2024-02-16T22:10:05.590 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-513.gb88fae66.el9.n 1/7 2024-02-16T22:10:05.590 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-513.gb88fae66.el9.noarch 2/7 2024-02-16T22:10:05.590 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-513.gb88fae66.el9.no 3/7 2024-02-16T22:10:05.590 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-16T22:10:05.590 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-16T22:10:05.590 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-16T22:10:05.682 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-16T22:10:05.682 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:05.683 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:05.903 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-16T22:10:05.903 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-16T22:10:05.903 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:05.903 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:05.903 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-16T22:10:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-16T22:10:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-16T22:10:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-16T22:10:05.904 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:05.904 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:06.127 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-16T22:10:06.474 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:11 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:06.641 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:06.642 INFO:teuthology.orchestra.run.smithi103.stdout:=========================================================================================== 2024-02-16T22:10:06.642 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:10:06.642 INFO:teuthology.orchestra.run.smithi103.stdout:=========================================================================================== 2024-02-16T22:10:06.642 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:06.642 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 7.4 M 2024-02-16T22:10:06.642 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-16T22:10:06.642 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:06.643 INFO:teuthology.orchestra.run.smithi103.stdout:=========================================================================================== 2024-02-16T22:10:06.644 INFO:teuthology.orchestra.run.smithi103.stdout:Install 12 Packages 2024-02-16T22:10:06.644 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:06.644 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 37 M 2024-02-16T22:10:06.644 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 209 M 2024-02-16T22:10:06.644 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:06.712 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:11 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:06.881 INFO:teuthology.orchestra.run.smithi103.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 164 kB/s | 32 kB 00:00 2024-02-16T22:10:06.885 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:06.886 INFO:teuthology.orchestra.run.smithi002.stdout:=========================================================================================== 2024-02-16T22:10:06.886 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:10:06.886 INFO:teuthology.orchestra.run.smithi002.stdout:=========================================================================================== 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 7.4 M 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-16T22:10:06.887 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:06.888 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:06.888 INFO:teuthology.orchestra.run.smithi002.stdout:=========================================================================================== 2024-02-16T22:10:06.888 INFO:teuthology.orchestra.run.smithi002.stdout:Install 12 Packages 2024-02-16T22:10:06.888 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:06.888 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 37 M 2024-02-16T22:10:06.888 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 209 M 2024-02-16T22:10:06.889 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:06.948 INFO:teuthology.orchestra.run.smithi103.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 257 kB/s | 17 kB 00:00 2024-02-16T22:10:07.031 INFO:teuthology.orchestra.run.smithi103.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 411 kB/s | 34 kB 00:00 2024-02-16T22:10:07.190 INFO:teuthology.orchestra.run.smithi103.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.9 MB/s | 3.0 MB 00:00 2024-02-16T22:10:07.355 INFO:teuthology.orchestra.run.smithi002.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 136 kB/s | 32 kB 00:00 2024-02-16T22:10:07.438 INFO:teuthology.orchestra.run.smithi002.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 205 kB/s | 17 kB 00:00 2024-02-16T22:10:07.522 INFO:teuthology.orchestra.run.smithi002.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 410 kB/s | 34 kB 00:00 2024-02-16T22:10:07.532 INFO:teuthology.orchestra.run.smithi103.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 9.5 MB/s | 4.7 MB 00:00 2024-02-16T22:10:07.624 INFO:teuthology.orchestra.run.smithi103.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-02-16T22:10:07.650 INFO:teuthology.orchestra.run.smithi103.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 217 kB 00:00 2024-02-16T22:10:07.739 INFO:teuthology.orchestra.run.smithi002.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-02-16T22:10:07.800 INFO:teuthology.orchestra.run.smithi103.stdout:(8/12): ceph-mgr-diskprediction-local-17.2.7-51 6.6 MB/s | 7.4 MB 00:01 2024-02-16T22:10:08.072 INFO:teuthology.orchestra.run.smithi002.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 8.6 MB/s | 4.7 MB 00:00 2024-02-16T22:10:08.200 INFO:teuthology.orchestra.run.smithi103.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 802 kB 00:00 2024-02-16T22:10:08.223 INFO:teuthology.orchestra.run.smithi002.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.9 MB/s | 4.8 MB 00:00 2024-02-16T22:10:08.248 INFO:teuthology.orchestra.run.smithi002.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 217 kB 00:00 2024-02-16T22:10:08.384 INFO:teuthology.orchestra.run.smithi103.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-02-16T22:10:08.409 INFO:teuthology.orchestra.run.smithi103.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 320 kB/s | 194 kB 00:00 2024-02-16T22:10:08.565 INFO:teuthology.orchestra.run.smithi002.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 802 kB 00:00 2024-02-16T22:10:08.676 INFO:teuthology.orchestra.run.smithi103.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 519 kB/s | 245 kB 00:00 2024-02-16T22:10:08.678 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:08.679 INFO:teuthology.orchestra.run.smithi103.stdout:Total 18 MB/s | 37 MB 00:02 2024-02-16T22:10:08.698 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:08.724 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:08.724 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:08.756 INFO:teuthology.orchestra.run.smithi002.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-02-16T22:10:08.782 INFO:teuthology.orchestra.run.smithi002.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 905 kB/s | 194 kB 00:00 2024-02-16T22:10:08.999 INFO:teuthology.orchestra.run.smithi002.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-02-16T22:10:09.149 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:09.150 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:09.849 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:09.924 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-16T22:10:09.955 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-16T22:10:10.073 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-16T22:10:10.108 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-16T22:10:10.288 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-16T22:10:10.316 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-16T22:10:10.392 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-16T22:10:11.051 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-16T22:10:11.153 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-16T22:10:11.965 INFO:teuthology.orchestra.run.smithi002.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 1.5 MB/s | 7.4 MB 00:04 2024-02-16T22:10:11.968 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:11.969 INFO:teuthology.orchestra.run.smithi002.stdout:Total 7.3 MB/s | 37 MB 00:05 2024-02-16T22:10:11.991 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:12.019 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:12.019 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:12.390 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-16T22:10:12.453 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:12.454 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:13.284 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-16T22:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-16T22:10:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-16T22:10:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-513.gb88fae 12/12 2024-02-16T22:10:13.494 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-16T22:10:13.648 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-16T22:10:13.837 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-16T22:10:13.872 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-16T22:10:13.951 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-16T22:10:14.641 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-16T22:10:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-16T22:10:16.033 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-513.gb88fae 12/12 2024-02-16T22:10:16.033 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-513.gb88fae 1/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-16T22:10:16.034 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-16T22:10:16.062 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-16T22:10:16.463 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-16T22:10:16.464 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-16T22:10:16.464 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-16T22:10:16.464 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-16T22:10:16.464 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:16.464 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:16.803 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-rook 2024-02-16T22:10:17.141 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-16T22:10:17.171 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-513.gb88fae 12/12 2024-02-16T22:10:17.391 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:22 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:17.559 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:17.560 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-02-16T22:10:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:10:17.560 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-02-16T22:10:17.560 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 47 k 2024-02-16T22:10:17.560 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-16T22:10:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-16T22:10:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:17.561 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-02-16T22:10:17.562 INFO:teuthology.orchestra.run.smithi103.stdout:Install 13 Packages 2024-02-16T22:10:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:17.562 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.0 M 2024-02-16T22:10:17.562 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 26 M 2024-02-16T22:10:17.562 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:17.879 INFO:teuthology.orchestra.run.smithi103.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 128 kB/s | 18 kB 00:00 2024-02-16T22:10:17.895 INFO:teuthology.orchestra.run.smithi103.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 154 kB/s | 24 kB 00:00 2024-02-16T22:10:17.962 INFO:teuthology.orchestra.run.smithi103.stdout:(3/13): ceph-mgr-rook-17.2.7-513.gb88fae66.el9. 210 kB/s | 47 kB 00:00 2024-02-16T22:10:18.004 INFO:teuthology.orchestra.run.smithi103.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 174 kB 00:00 2024-02-16T22:10:18.030 INFO:teuthology.orchestra.run.smithi103.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 986 kB/s | 132 kB 00:00 2024-02-16T22:10:18.063 INFO:teuthology.orchestra.run.smithi103.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 211 kB 00:00 2024-02-16T22:10:18.080 INFO:teuthology.orchestra.run.smithi103.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 567 kB/s | 43 kB 00:00 2024-02-16T22:10:18.197 INFO:teuthology.orchestra.run.smithi103.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-02-16T22:10:18.222 INFO:teuthology.orchestra.run.smithi103.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-02-16T22:10:18.498 INFO:teuthology.orchestra.run.smithi103.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 215 kB/s | 59 kB 00:00 2024-02-16T22:10:18.856 INFO:teuthology.orchestra.run.smithi103.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 250 kB/s | 90 kB 00:00 2024-02-16T22:10:19.357 INFO:teuthology.orchestra.run.smithi103.stdout:(12/13): python3-google-auth-2.27.0-1.el9.noarc 159 kB/s | 203 kB 00:01 2024-02-16T22:10:19.761 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-513.gb88fae 12/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-513.gb88fae 1/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-16T22:10:19.762 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-16T22:10:20.232 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-16T22:10:20.232 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:20.232 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:20.232 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:20.233 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:20.602 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-rook 2024-02-16T22:10:21.192 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:26 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:21.367 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:21.368 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================= 2024-02-16T22:10:21.368 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:10:21.369 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================= 2024-02-16T22:10:21.369 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:21.369 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 47 k 2024-02-16T22:10:21.369 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-16T22:10:21.369 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-16T22:10:21.369 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-16T22:10:21.369 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-16T22:10:21.369 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-16T22:10:21.369 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================= 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout:Install 13 Packages 2024-02-16T22:10:21.370 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:21.371 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 2.0 M 2024-02-16T22:10:21.371 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 26 M 2024-02-16T22:10:21.371 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:21.672 INFO:teuthology.orchestra.run.smithi002.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 110 kB/s | 24 kB 00:00 2024-02-16T22:10:21.697 INFO:teuthology.orchestra.run.smithi002.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 73 kB/s | 18 kB 00:00 2024-02-16T22:10:21.872 INFO:teuthology.orchestra.run.smithi002.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 877 kB/s | 174 kB 00:00 2024-02-16T22:10:21.906 INFO:teuthology.orchestra.run.smithi002.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 634 kB/s | 132 kB 00:00 2024-02-16T22:10:21.973 INFO:teuthology.orchestra.run.smithi002.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.1 MB/s | 211 kB 00:00 2024-02-16T22:10:21.990 INFO:teuthology.orchestra.run.smithi002.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 511 kB/s | 43 kB 00:00 2024-02-16T22:10:22.015 INFO:teuthology.orchestra.run.smithi002.stdout:(7/13): ceph-mgr-rook-17.2.7-513.gb88fae66.el9. 84 kB/s | 47 kB 00:00 2024-02-16T22:10:22.240 INFO:teuthology.orchestra.run.smithi002.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-02-16T22:10:22.266 INFO:teuthology.orchestra.run.smithi002.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-02-16T22:10:22.449 INFO:teuthology.orchestra.run.smithi002.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 323 kB/s | 59 kB 00:00 2024-02-16T22:10:22.508 INFO:teuthology.orchestra.run.smithi002.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 412 kB/s | 203 kB 00:00 2024-02-16T22:10:22.550 INFO:teuthology.orchestra.run.smithi002.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 895 kB/s | 90 kB 00:00 2024-02-16T22:10:23.125 INFO:teuthology.orchestra.run.smithi002.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.2 MB/s | 1.0 MB 00:00 2024-02-16T22:10:23.127 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:23.127 INFO:teuthology.orchestra.run.smithi002.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-02-16T22:10:23.128 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:23.144 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:23.144 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:23.256 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:23.258 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:23.822 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:23.996 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-16T22:10:24.065 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-16T22:10:24.125 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-16T22:10:24.161 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-16T22:10:24.203 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-16T22:10:24.309 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-16T22:10:24.428 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-16T22:10:24.480 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-16T22:10:25.238 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-16T22:10:25.300 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-16T22:10:25.334 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-16T22:10:25.381 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-16T22:10:25.404 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-rook-2:17.2.7-513.gb88fae66.el9.noarch 13/13 2024-02-16T22:10:25.598 INFO:teuthology.orchestra.run.smithi103.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 141 kB/s | 1.0 MB 00:07 2024-02-16T22:10:25.601 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:25.601 INFO:teuthology.orchestra.run.smithi103.stdout:Total 260 kB/s | 2.0 MB 00:08 2024-02-16T22:10:25.601 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:25.616 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:25.616 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:25.725 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:25.726 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-513.gb88fae66.el9.noarch 13/13 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-rook-2:17.2.7-513.gb88fae66.el9.noarch 1/13 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-16T22:10:26.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-16T22:10:26.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-16T22:10:26.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-16T22:10:26.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-16T22:10:26.244 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:26.422 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-16T22:10:26.483 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-16T22:10:26.547 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-16T22:10:26.583 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-16T22:10:26.583 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:26.583 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-16T22:10:26.584 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:26.585 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:26.585 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-16T22:10:26.626 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-16T22:10:26.719 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-16T22:10:26.806 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-mgr-cephadm 2024-02-16T22:10:26.829 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-16T22:10:26.877 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-16T22:10:27.392 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:32 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:27.567 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:27.568 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================= 2024-02-16T22:10:27.568 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================= 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 121 k 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:27.569 INFO:teuthology.orchestra.run.smithi002.stdout:======================================================================================= 2024-02-16T22:10:27.570 INFO:teuthology.orchestra.run.smithi002.stdout:Install 6 Packages 2024-02-16T22:10:27.570 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:27.570 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 6.8 M 2024-02-16T22:10:27.570 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 33 M 2024-02-16T22:10:27.570 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:27.602 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-16T22:10:27.658 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-16T22:10:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-16T22:10:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-16T22:10:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:17.2.7-513.gb88fae66.el9.noarch 13/13 2024-02-16T22:10:28.159 INFO:teuthology.orchestra.run.smithi002.stdout:(1/6): ceph-mgr-cephadm-17.2.7-513.gb88fae66.el 500 kB/s | 121 kB 00:00 2024-02-16T22:10:28.401 INFO:teuthology.orchestra.run.smithi002.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 469 kB/s | 227 kB 00:00 2024-02-16T22:10:28.493 INFO:teuthology.orchestra.run.smithi002.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 631 kB/s | 58 kB 00:00 2024-02-16T22:10:28.545 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-513.gb88fae66.el9.noarch 13/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:17.2.7-513.gb88fae66.el9.noarch 1/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-16T22:10:28.546 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-16T22:10:28.735 INFO:teuthology.orchestra.run.smithi002.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 211 kB/s | 51 kB 00:00 2024-02-16T22:10:28.902 INFO:teuthology.orchestra.run.smithi002.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.9 MB/s | 5.8 MB 00:00 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-16T22:10:28.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-16T22:10:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-16T22:10:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-16T22:10:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-16T22:10:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-16T22:10:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-16T22:10:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-16T22:10:28.926 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:28.926 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:29.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-mgr-cephadm 2024-02-16T22:10:29.739 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:34 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:29.905 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:17.2.7-513.gb88fae66.el9 ceph-noarch 121 k 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:29.906 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:29.907 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-02-16T22:10:29.907 INFO:teuthology.orchestra.run.smithi103.stdout:Install 6 Packages 2024-02-16T22:10:29.907 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:29.907 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 6.8 M 2024-02-16T22:10:29.907 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 33 M 2024-02-16T22:10:29.907 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:30.768 INFO:teuthology.orchestra.run.smithi103.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 437 kB/s | 227 kB 00:00 2024-02-16T22:10:31.018 INFO:teuthology.orchestra.run.smithi103.stdout:(2/6): ceph-mgr-cephadm-17.2.7-513.gb88fae66.el 158 kB/s | 121 kB 00:00 2024-02-16T22:10:31.152 INFO:teuthology.orchestra.run.smithi103.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.4 MB/s | 5.8 MB 00:00 2024-02-16T22:10:31.536 INFO:teuthology.orchestra.run.smithi103.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 112 kB/s | 58 kB 00:00 2024-02-16T22:10:31.586 INFO:teuthology.orchestra.run.smithi103.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 118 kB/s | 51 kB 00:00 2024-02-16T22:10:32.510 INFO:teuthology.orchestra.run.smithi002.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 126 kB/s | 548 kB 00:04 2024-02-16T22:10:32.513 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:32.513 INFO:teuthology.orchestra.run.smithi002.stdout:Total 1.4 MB/s | 6.8 MB 00:04 2024-02-16T22:10:32.513 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:32.524 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:32.524 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:32.623 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:32.623 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:32.962 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:33.135 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-16T22:10:33.188 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-16T22:10:33.739 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-16T22:10:33.833 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-16T22:10:33.896 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-16T22:10:33.924 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-513.gb88fae66.el9.noarch 6/6 2024-02-16T22:10:34.027 INFO:teuthology.orchestra.run.smithi103.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 168 kB/s | 548 kB 00:03 2024-02-16T22:10:34.030 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:34.030 INFO:teuthology.orchestra.run.smithi103.stdout:Total 1.6 MB/s | 6.8 MB 00:04 2024-02-16T22:10:34.031 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:34.041 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:34.041 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:34.135 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:34.135 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:34.466 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:34.634 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-16T22:10:34.696 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-16T22:10:34.704 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-513.gb88fae66.el9.noarch 6/6 2024-02-16T22:10:34.704 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/6 2024-02-16T22:10:34.704 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-16T22:10:34.704 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-16T22:10:34.704 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-16T22:10:34.705 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-16T22:10:35.024 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:35.212 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-16T22:10:35.286 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-fuse 2024-02-16T22:10:35.291 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-16T22:10:35.346 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-16T22:10:35.375 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-513.gb88fae66.el9.noarch 6/6 2024-02-16T22:10:35.872 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:40 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:36.045 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse x86_64 2:17.2.7-513.gb88fae66.el9 ceph 812 k 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:36.046 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:36.047 INFO:teuthology.orchestra.run.smithi002.stdout:Install 2 Packages 2024-02-16T22:10:36.047 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:36.047 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 890 k 2024-02-16T22:10:36.047 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 2.6 M 2024-02-16T22:10:36.047 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:36.116 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-513.gb88fae66.el9.noarch 6/6 2024-02-16T22:10:36.116 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-513.gb88fae66.el9.noarch 1/6 2024-02-16T22:10:36.117 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-16T22:10:36.117 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-16T22:10:36.117 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-16T22:10:36.117 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-16T22:10:36.362 INFO:teuthology.orchestra.run.smithi002.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 250 kB/s | 79 kB 00:00 2024-02-16T22:10:36.428 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-16T22:10:36.428 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:36.429 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:36.429 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:17.2.7-513.gb88fae66.el9.noarch 2024-02-16T22:10:36.429 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-16T22:10:36.429 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-16T22:10:36.429 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-16T22:10:36.429 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-16T22:10:36.429 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-16T22:10:36.429 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:36.429 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:36.683 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-fuse 2024-02-16T22:10:37.254 INFO:teuthology.orchestra.run.smithi002.stdout:(2/2): ceph-fuse-17.2.7-513.gb88fae66.el9.x86_6 673 kB/s | 812 kB 00:01 2024-02-16T22:10:37.255 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:37.255 INFO:teuthology.orchestra.run.smithi002.stdout:Total 737 kB/s | 890 kB 00:01 2024-02-16T22:10:37.256 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:37.256 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:42 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:37.280 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:37.281 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:37.363 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:37.364 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:37.422 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:37.422 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:17.2.7-513.gb88fae66.el9 ceph 812 k 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout:Install 2 Packages 2024-02-16T22:10:37.423 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:37.424 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 890 k 2024-02-16T22:10:37.424 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 2.6 M 2024-02-16T22:10:37.424 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:37.548 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:37.572 INFO:teuthology.orchestra.run.smithi103.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 532 kB/s | 79 kB 00:00 2024-02-16T22:10:37.694 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-16T22:10:37.764 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 2/2 2024-02-16T22:10:37.830 INFO:teuthology.orchestra.run.smithi103.stdout:(2/2): ceph-fuse-17.2.7-513.gb88fae66.el9.x86_6 2.0 MB/s | 812 kB 00:00 2024-02-16T22:10:37.832 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:37.832 INFO:teuthology.orchestra.run.smithi103.stdout:Total 2.1 MB/s | 890 kB 00:00 2024-02-16T22:10:37.832 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:37.852 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:37.852 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:37.926 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:37.926 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:38.111 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:38.244 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-16T22:10:38.321 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 2/2 2024-02-16T22:10:38.480 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 2/2 2024-02-16T22:10:38.480 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 1/2 2024-02-16T22:10:38.815 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-16T22:10:38.815 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:38.815 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:38.815 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-16T22:10:38.815 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:38.815 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:39.005 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 2/2 2024-02-16T22:10:39.005 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 1/2 2024-02-16T22:10:39.049 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install librados-devel 2024-02-16T22:10:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-16T22:10:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:39.336 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:39.336 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-16T22:10:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:39.337 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:39.606 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados-devel 2024-02-16T22:10:39.642 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:44 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:39.816 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:39.816 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel x86_64 2:17.2.7-513.gb88fae66.el9 ceph 126 k 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 126 k 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 448 k 2024-02-16T22:10:39.817 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:40.032 INFO:teuthology.orchestra.run.smithi002.stdout:librados-devel-17.2.7-513.gb88fae66.el9.x86_64. 588 kB/s | 126 kB 00:00 2024-02-16T22:10:40.033 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:40.033 INFO:teuthology.orchestra.run.smithi002.stdout:Total 584 kB/s | 126 kB 00:00 2024-02-16T22:10:40.033 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:40.039 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:40.039 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:40.096 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:40.096 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:40.208 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:45 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:40.249 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:40.358 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librados-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:40.373 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:17.2.7-513.gb88fae66.el9 ceph 126 k 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 126 k 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 448 k 2024-02-16T22:10:40.374 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:40.604 INFO:teuthology.orchestra.run.smithi103.stdout:librados-devel-17.2.7-513.gb88fae66.el9.x86_64. 551 kB/s | 126 kB 00:00 2024-02-16T22:10:40.605 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:40.605 INFO:teuthology.orchestra.run.smithi103.stdout:Total 547 kB/s | 126 kB 00:00 2024-02-16T22:10:40.605 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:40.610 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:40.610 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:40.665 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:40.666 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:40.813 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:41.124 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:41.124 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:41.124 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:41.124 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:41.124 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:41.125 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:41.337 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:41.408 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install libcephfs2 2024-02-16T22:10:41.662 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:41.662 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:41.663 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:41.663 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:41.663 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:41.663 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:41.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs2 2024-02-16T22:10:42.000 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:46 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:42.120 INFO:teuthology.orchestra.run.smithi002.stdout:Package libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:42.176 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:42.177 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-02-16T22:10:42.177 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:42.239 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install libcephfs-devel 2024-02-16T22:10:42.491 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:47 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:42.607 INFO:teuthology.orchestra.run.smithi103.stdout:Package libcephfs2-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:42.661 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:42.662 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-02-16T22:10:42.662 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:42.732 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install libcephfs-devel 2024-02-16T22:10:42.828 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:47 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:43.004 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:43.004 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel x86_64 2:17.2.7-513.gb88fae66.el9 ceph 25 k 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 25 k 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 110 k 2024-02-16T22:10:43.005 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:43.178 INFO:teuthology.orchestra.run.smithi002.stdout:libcephfs-devel-17.2.7-513.gb88fae66.el9.x86_64 145 kB/s | 25 kB 00:00 2024-02-16T22:10:43.178 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:43.179 INFO:teuthology.orchestra.run.smithi002.stdout:Total 143 kB/s | 25 kB 00:00 2024-02-16T22:10:43.179 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:43.182 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:43.182 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:43.184 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:43.184 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:43.259 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:43.325 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:48 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:43.369 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:43.504 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:43.504 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-02-16T22:10:43.504 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:43.504 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:17.2.7-513.gb88fae66.el9 ceph 25 k 2024-02-16T22:10:43.504 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:43.505 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:43.505 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:43.505 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-02-16T22:10:43.505 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:43.505 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 25 k 2024-02-16T22:10:43.505 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 110 k 2024-02-16T22:10:43.505 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:43.685 INFO:teuthology.orchestra.run.smithi103.stdout:libcephfs-devel-17.2.7-513.gb88fae66.el9.x86_64 139 kB/s | 25 kB 00:00 2024-02-16T22:10:43.686 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:43.686 INFO:teuthology.orchestra.run.smithi103.stdout:Total 137 kB/s | 25 kB 00:00 2024-02-16T22:10:43.686 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:43.690 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:43.690 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:43.692 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:43.692 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:43.748 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:43.772 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:44.022 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:44.022 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:44.022 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:44.022 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:44.022 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:44.022 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:44.205 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install librados2 2024-02-16T22:10:44.299 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:44.570 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:44.570 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:44.570 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:44.570 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:44.570 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:44.570 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:44.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librados2 2024-02-16T22:10:44.794 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:49 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:44.912 INFO:teuthology.orchestra.run.smithi002.stdout:Package librados2-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:44.969 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-02-16T22:10:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:45.030 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install librbd1 2024-02-16T22:10:45.337 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:50 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:45.451 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:45.502 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:45.503 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-02-16T22:10:45.503 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:45.568 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install librbd1 2024-02-16T22:10:45.618 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:50 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:45.737 INFO:teuthology.orchestra.run.smithi002.stdout:Package librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:45.792 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:45.793 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-02-16T22:10:45.793 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:45.867 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-rados 2024-02-16T22:10:46.148 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:51 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:46.263 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:46.315 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-02-16T22:10:46.316 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:46.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rados 2024-02-16T22:10:46.453 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:51 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:46.571 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-rados-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:46.627 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:46.628 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-02-16T22:10:46.628 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:46.692 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-rgw 2024-02-16T22:10:46.980 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:51 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:47.094 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rados-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:47.145 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-02-16T22:10:47.146 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:47.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rgw 2024-02-16T22:10:47.275 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:52 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:47.393 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-rgw-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:47.450 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:47.451 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-02-16T22:10:47.451 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:47.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-cephfs 2024-02-16T22:10:47.781 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:52 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:47.895 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rgw-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:47.947 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:47.948 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-02-16T22:10:47.948 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:48.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-cephfs 2024-02-16T22:10:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:53 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:48.223 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-cephfs-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:48.279 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:48.280 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-02-16T22:10:48.280 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:48.348 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install python3-rbd 2024-02-16T22:10:48.598 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:53 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:48.714 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-cephfs-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:48.765 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-02-16T22:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:48.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install python3-rbd 2024-02-16T22:10:48.944 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:53 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:49.064 INFO:teuthology.orchestra.run.smithi002.stdout:Package python3-rbd-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:49.118 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:49.120 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2024-02-16T22:10:49.120 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:49.176 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-fuse 2024-02-16T22:10:49.413 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:54 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:49.527 INFO:teuthology.orchestra.run.smithi103.stdout:Package python3-rbd-2:17.2.7-513.gb88fae66.el9.x86_64 is already installed. 2024-02-16T22:10:49.579 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-02-16T22:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:49.648 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-fuse 2024-02-16T22:10:49.761 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:54 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:49.934 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:49.935 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:49.935 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2024-02-16T22:10:49.935 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse x86_64 2:17.2.7-513.gb88fae66.el9 ceph 81 k 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 81 k 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 222 k 2024-02-16T22:10:49.936 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:50.225 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:55 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:50.395 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:17.2.7-513.gb88fae66.el9 ceph 81 k 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 81 k 2024-02-16T22:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 222 k 2024-02-16T22:10:50.397 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:50.464 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-fuse-17.2.7-513.gb88fae66.el9.x86_64.rpm 154 kB/s | 81 kB 00:00 2024-02-16T22:10:50.464 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:50.465 INFO:teuthology.orchestra.run.smithi002.stdout:Total 154 kB/s | 81 kB 00:00 2024-02-16T22:10:50.465 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:50.473 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:50.473 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:50.531 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:50.532 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:50.668 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:50.773 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:50.905 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-fuse-17.2.7-513.gb88fae66.el9.x86_64.rpm 160 kB/s | 81 kB 00:00 2024-02-16T22:10:50.906 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:50.906 INFO:teuthology.orchestra.run.smithi103.stdout:Total 159 kB/s | 81 kB 00:00 2024-02-16T22:10:50.906 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:50.913 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:50.913 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:50.968 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:50.969 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:51.100 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:51.187 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:51.494 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:51.494 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:51.494 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:51.494 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:51.494 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:51.494 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:51.598 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:51.727 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-mirror 2024-02-16T22:10:51.910 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:51.910 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:51.910 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:51.910 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:51.910 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:52.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-mirror 2024-02-16T22:10:52.310 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:01:57 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:52.485 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repo Size 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror x86_64 2:17.2.7-513.gb88fae66.el9 ceph 2.9 M 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-02-16T22:10:52.486 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:52.487 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 2.9 M 2024-02-16T22:10:52.487 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 12 M 2024-02-16T22:10:52.487 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:52.707 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:01:57 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:52.880 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:52.881 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:52.881 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repo Size 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:17.2.7-513.gb88fae66.el9 ceph 2.9 M 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 2.9 M 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 12 M 2024-02-16T22:10:52.882 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-mirror-17.2.7-513.gb88fae66.el9.x86_64.rpm 6.5 MB/s | 2.9 MB 00:00 2024-02-16T22:10:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:53.335 INFO:teuthology.orchestra.run.smithi103.stdout:Total 6.5 MB/s | 2.9 MB 00:00 2024-02-16T22:10:53.336 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:53.344 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:53.345 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:53.650 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-mirror-17.2.7-513.gb88fae66.el9.x86_64.rpm 2.5 MB/s | 2.9 MB 00:01 2024-02-16T22:10:53.651 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:53.651 INFO:teuthology.orchestra.run.smithi002.stdout:Total 2.5 MB/s | 2.9 MB 00:01 2024-02-16T22:10:53.651 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:53.662 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:53.662 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:53.738 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:53.738 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:53.825 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:53.894 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:53.919 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:53.919 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:10:53.919 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-16T22:10:53.919 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-16T22:10:53.919 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-16T22:10:53.919 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:54.170 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:54.242 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-mirror-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:54.266 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-mirror-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:54.266 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-16T22:10:54.266 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-16T22:10:54.266 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-16T22:10:54.267 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-16T22:10:54.267 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:55.061 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:55.061 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:55.061 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:55.061 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:55.061 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:55.290 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install rbd-nbd 2024-02-16T22:10:55.385 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-mirror-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:55.385 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:55.385 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:55.386 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:55.386 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:55.386 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:55.620 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install rbd-nbd 2024-02-16T22:10:55.882 INFO:teuthology.orchestra.run.smithi103.stdout:Last metadata expiration check: 0:02:00 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:56.050 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-02-16T22:10:56.051 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:56.051 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-02-16T22:10:56.051 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:56.051 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-02-16T22:10:56.051 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:17.2.7-513.gb88fae66.el9 ceph 165 k 2024-02-16T22:10:56.051 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:56.051 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-02-16T22:10:56.052 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-02-16T22:10:56.052 INFO:teuthology.orchestra.run.smithi103.stdout:Install 1 Package 2024-02-16T22:10:56.052 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:56.052 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 165 k 2024-02-16T22:10:56.052 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 477 k 2024-02-16T22:10:56.052 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-02-16T22:10:56.209 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:02:01 ago on Fri 16 Feb 2024 10:08:55 PM UTC. 2024-02-16T22:10:56.385 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-02-16T22:10:56.385 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd x86_64 2:17.2.7-513.gb88fae66.el9 ceph 165 k 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 165 k 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 477 k 2024-02-16T22:10:56.386 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-02-16T22:10:56.597 INFO:teuthology.orchestra.run.smithi103.stdout:rbd-nbd-17.2.7-513.gb88fae66.el9.x86_64.rpm 302 kB/s | 165 kB 00:00 2024-02-16T22:10:56.598 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:56.598 INFO:teuthology.orchestra.run.smithi103.stdout:Total 301 kB/s | 165 kB 00:00 2024-02-16T22:10:56.598 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-02-16T22:10:56.606 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-02-16T22:10:56.606 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-02-16T22:10:56.613 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-nbd-17.2.7-513.gb88fae66.el9.x86_64.rpm 727 kB/s | 165 kB 00:00 2024-02-16T22:10:56.613 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:10:56.614 INFO:teuthology.orchestra.run.smithi002.stdout:Total 722 kB/s | 165 kB 00:00 2024-02-16T22:10:56.614 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-02-16T22:10:56.622 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-02-16T22:10:56.622 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-02-16T22:10:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-02-16T22:10:56.662 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-02-16T22:10:56.680 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-02-16T22:10:56.680 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-02-16T22:10:56.816 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-02-16T22:10:56.830 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-02-16T22:10:56.911 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:56.923 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-nbd-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:57.320 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-nbd-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:57.393 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-nbd-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:57.608 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:57.608 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:57.608 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-02-16T22:10:57.608 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:57.608 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:10:57.608 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-02-16T22:10:57.694 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-nbd-2:17.2.7-513.gb88fae66.el9.x86_64 1/1 2024-02-16T22:10:57.694 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:57.694 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-02-16T22:10:57.694 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd-2:17.2.7-513.gb88fae66.el9.x86_64 2024-02-16T22:10:57.694 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:10:57.694 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-02-16T22:10:57.835 DEBUG:teuthology.parallel:result is None 2024-02-16T22:10:57.911 DEBUG:teuthology.parallel:result is None 2024-02-16T22:10:57.912 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-16T22:10:57.912 INFO:teuthology.packaging:ref: None 2024-02-16T22:10:57.912 INFO:teuthology.packaging:tag: None 2024-02-16T22:10:57.912 INFO:teuthology.packaging:branch: quincy 2024-02-16T22:10:57.912 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:10:57.912 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-16T22:10:58.284 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-16T22:10:58.304 INFO:teuthology.orchestra.run.smithi002.stdout:17.2.7-513.gb88fae66.el9 2024-02-16T22:10:58.305 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-513.gb88fae66.el9 2024-02-16T22:10:58.305 INFO:teuthology.task.install:The correct ceph version 17.2.7-513.gb88fae66 is installed. 2024-02-16T22:10:58.307 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-16T22:10:58.307 INFO:teuthology.packaging:ref: None 2024-02-16T22:10:58.307 INFO:teuthology.packaging:tag: None 2024-02-16T22:10:58.307 INFO:teuthology.packaging:branch: quincy 2024-02-16T22:10:58.307 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:10:58.307 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-16T22:10:58.693 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-16T22:10:58.716 INFO:teuthology.orchestra.run.smithi103.stdout:17.2.7-513.gb88fae66.el9 2024-02-16T22:10:58.716 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-513.gb88fae66.el9 2024-02-16T22:10:58.716 INFO:teuthology.task.install:The correct ceph version 17.2.7-513.gb88fae66 is installed. 2024-02-16T22:10:58.718 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-16T22:10:58.718 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:10:58.718 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-16T22:10:58.747 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:10:58.747 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-16T22:10:58.786 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-16T22:10:58.786 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:10:58.786 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2024-02-16T22:10:58.815 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-16T22:10:58.883 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:10:58.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-02-16T22:10:58.912 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-16T22:10:58.980 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-16T22:10:58.981 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:10:58.981 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-16T22:10:59.008 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-16T22:10:59.075 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:10:59.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-16T22:10:59.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-16T22:10:59.171 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-16T22:10:59.172 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:10:59.172 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2024-02-16T22:10:59.198 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-16T22:10:59.264 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:10:59.265 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-02-16T22:10:59.292 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-16T22:10:59.359 INFO:teuthology.run_tasks:Running task print... 2024-02-16T22:10:59.369 INFO:teuthology.task.print:**** done install task... 2024-02-16T22:10:59.369 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-16T22:10:59.469 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'} 2024-02-16T22:10:59.469 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-02-16T22:10:59.470 INFO:tasks.cephadm:Cluster fsid is 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:10:59.470 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-16T22:10:59.470 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-16T22:10:59.470 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi002': '172.21.15.2', 'mon.smithi103': '172.21.15.103'} 2024-02-16T22:10:59.470 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-16T22:10:59.470 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname $(hostname -s) 2024-02-16T22:10:59.498 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-02-16T22:10:59.525 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-02-16T22:10:59.526 DEBUG:teuthology.orchestra.run.smithi002:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-16T22:10:59.782 INFO:teuthology.orchestra.run.smithi002.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 16 22:10 /home/ubuntu/cephtest/cephadm 2024-02-16T22:10:59.782 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-16T22:11:00.031 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 16 22:11 /home/ubuntu/cephtest/cephadm 2024-02-16T22:11:00.032 DEBUG:teuthology.orchestra.run.smithi002:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-16T22:11:00.048 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-16T22:11:00.073 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-02-16T22:11:00.073 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-16T22:11:00.090 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-16T22:11:00.969 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-16T22:11:00.999 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-16T22:11:31.696 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-02-16T22:11:31.696 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)", 2024-02-16T22:11:31.697 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "870a0944ccb83fae8829e20e516a8aeec00e19c6a392cc9e32528216b9cb93f0", 2024-02-16T22:11:31.697 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-02-16T22:11:31.697 INFO:teuthology.orchestra.run.smithi103.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c" 2024-02-16T22:11:31.697 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-02-16T22:11:31.697 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-02-16T22:11:38.965 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-16T22:11:38.965 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)", 2024-02-16T22:11:38.966 INFO:teuthology.orchestra.run.smithi002.stdout: "image_id": "870a0944ccb83fae8829e20e516a8aeec00e19c6a392cc9e32528216b9cb93f0", 2024-02-16T22:11:38.966 INFO:teuthology.orchestra.run.smithi002.stdout: "repo_digests": [ 2024-02-16T22:11:38.966 INFO:teuthology.orchestra.run.smithi002.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c" 2024-02-16T22:11:38.966 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-02-16T22:11:38.966 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-16T22:11:38.986 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph 2024-02-16T22:11:39.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-02-16T22:11:39.071 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /etc/ceph 2024-02-16T22:11:39.098 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-02-16T22:11:39.141 INFO:tasks.cephadm:Writing seed config... 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-16T22:11:39.142 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-16T22:11:39.143 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-16T22:11:39.144 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-16T22:11:39.145 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:11:39.145 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-16T22:11:39.160 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 43c11c36-cd18-11ee-95ba-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-16T22:11:39.160 DEBUG:teuthology.orchestra.run.smithi002:mon.smithi002> sudo journalctl -f -n 0 -u ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mon.smithi002.service 2024-02-16T22:11:39.203 INFO:tasks.cephadm:Bootstrapping... 2024-02-16T22:11:39.203 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-16T22:11:39.412 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-02-16T22:11:39.412 INFO:teuthology.orchestra.run.smithi002.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '43c11c36-cd18-11ee-95ba-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.2', '--skip-admin-label'] 2024-02-16T22:11:39.437 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-16T22:11:39.437 INFO:teuthology.orchestra.run.smithi002.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-16T22:11:39.438 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying podman|docker is present... 2024-02-16T22:11:39.461 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-16T22:11:39.462 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying lvm2 is present... 2024-02-16T22:11:39.462 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying time synchronization is in place... 2024-02-16T22:11:39.469 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-16T22:11:39.470 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-16T22:11:39.476 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-16T22:11:39.476 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-02-16T22:11:39.482 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2024-02-16T22:11:39.489 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2024-02-16T22:11:39.489 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2024-02-16T22:11:39.489 INFO:teuthology.orchestra.run.smithi002.stdout:Repeating the final host check... 2024-02-16T22:11:39.512 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-16T22:11:39.512 INFO:teuthology.orchestra.run.smithi002.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-16T22:11:39.512 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl is present 2024-02-16T22:11:39.512 INFO:teuthology.orchestra.run.smithi002.stdout:lvcreate is present 2024-02-16T22:11:39.519 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-16T22:11:39.519 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-16T22:11:39.524 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-16T22:11:39.524 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-02-16T22:11:39.531 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2024-02-16T22:11:39.537 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2024-02-16T22:11:39.537 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2024-02-16T22:11:39.538 INFO:teuthology.orchestra.run.smithi002.stdout:Host looks OK 2024-02-16T22:11:39.538 INFO:teuthology.orchestra.run.smithi002.stdout:Cluster fsid: 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:11:39.538 INFO:teuthology.orchestra.run.smithi002.stdout:Acquiring lock 139629903280640 on /run/cephadm/43c11c36-cd18-11ee-95ba-87774f69a715.lock 2024-02-16T22:11:39.538 INFO:teuthology.orchestra.run.smithi002.stdout:Lock 139629903280640 acquired on /run/cephadm/43c11c36-cd18-11ee-95ba-87774f69a715.lock 2024-02-16T22:11:39.538 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 3300 ... 2024-02-16T22:11:39.538 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 6789 ... 2024-02-16T22:11:39.538 INFO:teuthology.orchestra.run.smithi002.stdout:Base mon IP(s) is [172.21.15.2:3300, 172.21.15.2:6789], mon addrv is [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-02-16T22:11:39.541 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.2 metric 100 2024-02-16T22:11:39.542 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.2 metric 100 2024-02-16T22:11:39.544 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-16T22:11:39.544 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-16T22:11:39.545 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-16T22:11:39.547 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-16T22:11:39.547 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-16T22:11:39.547 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-16T22:11:39.547 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-16T22:11:39.547 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-02-16T22:11:39.547 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-16T22:11:39.548 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2024-02-16T22:11:39.548 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2024-02-16T22:11:39.548 INFO:teuthology.orchestra.run.smithi002.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-16T22:11:39.548 INFO:teuthology.orchestra.run.smithi002.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-16T22:11:39.549 INFO:teuthology.orchestra.run.smithi002.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-16T22:11:40.241 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 870a0944ccb83fae8829e20e516a8aeec00e19c6a392cc9e32528216b9cb93f0 2024-02-16T22:11:40.241 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-16T22:11:40.241 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Getting image source signatures 2024-02-16T22:11:40.241 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:5dfd87a714924b39df09034baf76a9bd7605cb8f0d34cc8ca301fad8be327611 2024-02-16T22:11:40.241 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-16T22:11:40.241 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying config sha256:870a0944ccb83fae8829e20e516a8aeec00e19c6a392cc9e32528216b9cb93f0 2024-02-16T22:11:40.241 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-16T22:11:41.104 INFO:teuthology.orchestra.run.smithi002.stdout:ceph: stdout ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable) 2024-02-16T22:11:41.104 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph version: ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable) 2024-02-16T22:11:41.104 INFO:teuthology.orchestra.run.smithi002.stdout:Extracting ceph user uid/gid from container image... 2024-02-16T22:11:41.960 INFO:teuthology.orchestra.run.smithi002.stdout:stat: stdout 167 167 2024-02-16T22:11:41.960 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial keys... 2024-02-16T22:11:42.841 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQCe3c9lKpnTDhAAW+EomIHIXzb4/YA2YU43SQ== 2024-02-16T22:11:47.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQCf3c9ly+K7CBAA4frxSrYoTqvcR541BTz5iA== 2024-02-16T22:11:56.125 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQCl3c9l42+MNBAAmqIF4IJp0z3+O4CEoVpvsA== 2024-02-16T22:11:56.126 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial monmap... 2024-02-16T22:11:56.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-16T22:11:56.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-02-16T22:11:56.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:11:56.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-16T22:11:56.983 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool for smithi002 [v2:172.21.15.2:3300,v1:172.21.15.2:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-16T22:11:56.983 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = octopus 2024-02-16T22:11:56.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: set fsid to 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:11:56.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-16T22:11:56.984 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:11:56.984 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mon... 2024-02-16T22:11:57.930 INFO:teuthology.orchestra.run.smithi002.stdout:create mon.smithi002 on 2024-02-16T22:11:58.128 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-16T22:11:58.273 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-16T22:11:58.423 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-43c11c36-cd18-11ee-95ba-87774f69a715.target → /etc/systemd/system/ceph-43c11c36-cd18-11ee-95ba-87774f69a715.target. 2024-02-16T22:11:58.423 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-43c11c36-cd18-11ee-95ba-87774f69a715.target → /etc/systemd/system/ceph-43c11c36-cd18-11ee-95ba-87774f69a715.target. 2024-02-16T22:11:58.589 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mon.smithi002 2024-02-16T22:11:58.589 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mon.smithi002.service: Unit ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mon.smithi002.service not loaded. 2024-02-16T22:11:58.748 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-43c11c36-cd18-11ee-95ba-87774f69a715.target.wants/ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mon.smithi002.service → /etc/systemd/system/ceph-43c11c36-cd18-11ee-95ba-87774f69a715@.service. 2024-02-16T22:11:59.273 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:11:59 smithi002 podman[28762]: 2024-02-16 22:11:59.216396831 +0000 UTC m=+0.350434547 container start a0370ab274c87a3987b97ed2ea3234edd40ac2a3013ef87e7fc01d8194cd4f85 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-b88fae6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-02-16T22:11:59.273 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:11:59 smithi002 bash[28762]: a0370ab274c87a3987b97ed2ea3234edd40ac2a3013ef87e7fc01d8194cd4f85 2024-02-16T22:11:59.286 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-16T22:11:59.286 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-02-16T22:11:59.294 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-16T22:11:59.294 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-02-16T22:11:59.294 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-02-16T22:11:59.294 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2024-02-16T22:11:59.294 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon to start... 2024-02-16T22:11:59.294 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon... 2024-02-16T22:11:59.527 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:11:59 smithi002 systemd[1]: Started Ceph mon.smithi002 for 43c11c36-cd18-11ee-95ba-87774f69a715. 2024-02-16T22:11:59.527 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:11:59 smithi002 ceph-mon[28780]: mkfs 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:11:59.527 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:11:59 smithi002 ceph-mon[28780]: mon.smithi002 is new leader, mons smithi002 in quorum (ranks 0) 2024-02-16T22:12:00.329 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout cluster: 2024-02-16T22:12:00.329 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout id: 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout services: 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi002 (age 0.517217s) 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout data: 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pgs: 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.712+0000 7f4b4940e700 1 Processor -- start 2024-02-16T22:12:00.330 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.712+0000 7f4b4940e700 1 -- start start 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.712+0000 7f4b4940e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.712+0000 7f4b4940e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b44102600 con 0x7f4b44101cb0 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.713+0000 7f4b42ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.713+0000 7f4b42ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52296/0 (socket says 172.21.15.2:52296) 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.713+0000 7f4b42ffd700 1 -- 172.21.15.2:0/980080414 learned_addr learned my addr 172.21.15.2:0/980080414 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.713+0000 7f4b42ffd700 1 -- 172.21.15.2:0/980080414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b44102740 con 0x7f4b44101cb0 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.713+0000 7f4b42ffd700 1 --2- 172.21.15.2:0/980080414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441020c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400d0d0 tx=0x7f4b3400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e711cbd2ea5757 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.713+0000 7f4b41ffb700 1 -- 172.21.15.2:0/980080414 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b3400ddf0 con 0x7f4b44101cb0 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.713+0000 7f4b41ffb700 1 -- 172.21.15.2:0/980080414 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3400b0d0 con 0x7f4b44101cb0 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.713+0000 7f4b41ffb700 1 -- 172.21.15.2:0/980080414 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b3400b3e0 con 0x7f4b44101cb0 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.714+0000 7f4b4940e700 1 -- 172.21.15.2:0/980080414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 msgr2=0x7f4b441020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.714+0000 7f4b4940e700 1 --2- 172.21.15.2:0/980080414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441020c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400d0d0 tx=0x7f4b3400ba50 comp rx=0 tx=0).stop 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.714+0000 7f4b4940e700 1 -- 172.21.15.2:0/980080414 shutdown_connections 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.714+0000 7f4b4940e700 1 --2- 172.21.15.2:0/980080414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441020c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:00.331 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.714+0000 7f4b4940e700 1 -- 172.21.15.2:0/980080414 >> 172.21.15.2:0/980080414 conn(0x7f4b440fc580 msgr2=0x7f4b440fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.714+0000 7f4b4940e700 1 -- 172.21.15.2:0/980080414 shutdown_connections 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.714+0000 7f4b4940e700 1 -- 172.21.15.2:0/980080414 wait complete. 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b4940e700 1 Processor -- start 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b4940e700 1 -- start start 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b4940e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441132b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b4940e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b441137f0 con 0x7f4b44101cb0 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b42ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b42ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441132b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52298/0 (socket says 172.21.15.2:52298) 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b42ffd700 1 -- 172.21.15.2:0/948024564 learned_addr learned my addr 172.21.15.2:0/948024564 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b42ffd700 1 -- 172.21.15.2:0/948024564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3400bcf0 con 0x7f4b44101cb0 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b42ffd700 1 --2- 172.21.15.2:0/948024564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441132b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4b34000f80 tx=0x7f4b34019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b3b7fe700 1 -- 172.21.15.2:0/948024564 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b34009c20 con 0x7f4b44101cb0 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b3b7fe700 1 -- 172.21.15.2:0/948024564 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b34009d80 con 0x7f4b44101cb0 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b4940e700 1 -- 172.21.15.2:0/948024564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b441139f0 con 0x7f4b44101cb0 2024-02-16T22:12:00.332 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.715+0000 7f4b3b7fe700 1 -- 172.21.15.2:0/948024564 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b340145b0 con 0x7f4b44101cb0 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.716+0000 7f4b4940e700 1 -- 172.21.15.2:0/948024564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b44114690 con 0x7f4b44101cb0 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.716+0000 7f4b3b7fe700 1 -- 172.21.15.2:0/948024564 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4b34014760 con 0x7f4b44101cb0 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.716+0000 7f4b3b7fe700 1 -- 172.21.15.2:0/948024564 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4b34028450 con 0x7f4b44101cb0 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.716+0000 7f4b4940e700 1 -- 172.21.15.2:0/948024564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4405f5c0 con 0x7f4b44101cb0 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.719+0000 7f4b3b7fe700 1 -- 172.21.15.2:0/948024564 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4b3402d060 con 0x7f4b44101cb0 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.790+0000 7f4b4940e700 1 -- 172.21.15.2:0/948024564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4b4404bed0 con 0x7f4b44101cb0 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.790+0000 7f4b3b7fe700 1 -- 172.21.15.2:0/948024564 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f4b34023070 con 0x7f4b44101cb0 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.791+0000 7f4b4940e700 1 -- 172.21.15.2:0/948024564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 msgr2=0x7f4b441132b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.791+0000 7f4b4940e700 1 --2- 172.21.15.2:0/948024564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441132b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4b34000f80 tx=0x7f4b34019000 comp rx=0 tx=0).stop 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.791+0000 7f4b4940e700 1 -- 172.21.15.2:0/948024564 shutdown_connections 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.791+0000 7f4b4940e700 1 --2- 172.21.15.2:0/948024564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b44101cb0 0x7f4b441132b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.791+0000 7f4b4940e700 1 -- 172.21.15.2:0/948024564 >> 172.21.15.2:0/948024564 conn(0x7f4b440fc580 msgr2=0x7f4b44105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.791+0000 7f4b4940e700 1 -- 172.21.15.2:0/948024564 shutdown_connections 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:11:59.791+0000 7f4b4940e700 1 -- 172.21.15.2:0/948024564 wait complete. 2024-02-16T22:12:00.333 INFO:teuthology.orchestra.run.smithi002.stdout:mon is available 2024-02-16T22:12:00.334 INFO:teuthology.orchestra.run.smithi002.stdout:Assimilating anything we can from ceph.conf... 2024-02-16T22:12:00.450 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:00 smithi002 ceph-mon[28780]: mon.smithi002 is new leader, mons smithi002 in quorum (ranks 0) 2024-02-16T22:12:00.450 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:00 smithi002 ceph-mon[28780]: monmap e1: 1 mons at {smithi002=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-16T22:12:00.450 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:00 smithi002 ceph-mon[28780]: fsmap 2024-02-16T22:12:00.450 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:00 smithi002 ceph-mon[28780]: osdmap e1: 0 total, 0 up, 0 in 2024-02-16T22:12:00.451 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:00 smithi002 ceph-mon[28780]: mgrmap e1: no daemons active 2024-02-16T22:12:00.451 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:00 smithi002 ceph-mon[28780]: from='client.? 172.21.15.2:0/948024564' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-16T22:12:01.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:01 smithi002 ceph-mon[28780]: from='client.? 172.21.15.2:0/2355800109' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-16T22:12:01.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:01 smithi002 ceph-mon[28780]: from='client.? 172.21.15.2:0/2355800109' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:02.185 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.766+0000 7f6e75fd7700 1 Processor -- start 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.766+0000 7f6e75fd7700 1 -- start start 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.766+0000 7f6e75fd7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70071480 0x7f6e70105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.766+0000 7f6e75fd7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e701058c0 con 0x7f6e70071480 2024-02-16T22:12:02.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.767+0000 7f6e6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70071480 0x7f6e70105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.767+0000 7f6e6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70071480 0x7f6e70105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59258/0 (socket says 172.21.15.2:59258) 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.767+0000 7f6e6f7fe700 1 -- 172.21.15.2:0/2799197505 learned_addr learned my addr 172.21.15.2:0/2799197505 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.767+0000 7f6e6f7fe700 1 -- 172.21.15.2:0/2799197505 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e70105a00 con 0x7f6e70071480 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.767+0000 7f6e6f7fe700 1 --2- 172.21.15.2:0/2799197505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70071480 0x7f6e70105380 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6e600098e0 tx=0x7f6e60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6cc21fdcc69961e server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.767+0000 7f6e6e7fc700 1 -- 172.21.15.2:0/2799197505 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e6000b860 con 0x7f6e70071480 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.767+0000 7f6e6e7fc700 1 -- 172.21.15.2:0/2799197505 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6000b9c0 con 0x7f6e70071480 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.767+0000 7f6e6e7fc700 1 -- 172.21.15.2:0/2799197505 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e6000bc90 con 0x7f6e70071480 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.768+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2799197505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70071480 msgr2=0x7f6e70105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.768+0000 7f6e75fd7700 1 --2- 172.21.15.2:0/2799197505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70071480 0x7f6e70105380 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6e600098e0 tx=0x7f6e60009250 comp rx=0 tx=0).stop 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.768+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2799197505 shutdown_connections 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.768+0000 7f6e75fd7700 1 --2- 172.21.15.2:0/2799197505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70071480 0x7f6e70105380 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.768+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2799197505 >> 172.21.15.2:0/2799197505 conn(0x7f6e700fcdd0 msgr2=0x7f6e700ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.768+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2799197505 shutdown_connections 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.768+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2799197505 wait complete. 2024-02-16T22:12:02.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.768+0000 7f6e75fd7700 1 Processor -- start 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e75fd7700 1 -- start start 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e75fd7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70110ec0 0x7f6e701112d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e75fd7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e70111810 con 0x7f6e70110ec0 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70110ec0 0x7f6e701112d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70110ec0 0x7f6e701112d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59272/0 (socket says 172.21.15.2:59272) 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e6f7fe700 1 -- 172.21.15.2:0/2355800109 learned_addr learned my addr 172.21.15.2:0/2355800109 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e6f7fe700 1 -- 172.21.15.2:0/2355800109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e600094f0 con 0x7f6e70110ec0 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e6f7fe700 1 --2- 172.21.15.2:0/2355800109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70110ec0 0x7f6e701112d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009290 tx=0x7f6e60016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e57fff700 1 -- 172.21.15.2:0/2355800109 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e6000bc90 con 0x7f6e70110ec0 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.769+0000 7f6e57fff700 1 -- 172.21.15.2:0/2355800109 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60020070 con 0x7f6e70110ec0 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.770+0000 7f6e57fff700 1 -- 172.21.15.2:0/2355800109 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e6001b5d0 con 0x7f6e70110ec0 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.770+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2355800109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70111a10 con 0x7f6e70110ec0 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.770+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2355800109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e70112520 con 0x7f6e70110ec0 2024-02-16T22:12:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.770+0000 7f6e57fff700 1 -- 172.21.15.2:0/2355800109 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6e60022030 con 0x7f6e70110ec0 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.770+0000 7f6e57fff700 1 -- 172.21.15.2:0/2355800109 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6e6001c7a0 con 0x7f6e70110ec0 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.771+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2355800109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7005f5c0 con 0x7f6e70110ec0 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.773+0000 7f6e57fff700 1 -- 172.21.15.2:0/2355800109 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f6e6001bb40 con 0x7f6e70110ec0 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.842+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2355800109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6e7004bf50 con 0x7f6e70110ec0 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.845+0000 7f6e57fff700 1 -- 172.21.15.2:0/2355800109 <== mon.0 v2:172.21.15.2:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6e6001bce0 con 0x7f6e70110ec0 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.846+0000 7f6e57fff700 1 -- 172.21.15.2:0/2355800109 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7f6e7004bf50 con 0x7f6e70110ec0 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.846+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2355800109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70110ec0 msgr2=0x7f6e701112d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.846+0000 7f6e75fd7700 1 --2- 172.21.15.2:0/2355800109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70110ec0 0x7f6e701112d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6e60009290 tx=0x7f6e60016000 comp rx=0 tx=0).stop 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.846+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2355800109 shutdown_connections 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.846+0000 7f6e75fd7700 1 --2- 172.21.15.2:0/2355800109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e70110ec0 0x7f6e701112d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.846+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2355800109 >> 172.21.15.2:0/2355800109 conn(0x7f6e700fcdd0 msgr2=0x7f6e700fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.847+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2355800109 shutdown_connections 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:00.847+0000 7f6e75fd7700 1 -- 172.21.15.2:0/2355800109 wait complete. 2024-02-16T22:12:02.189 INFO:teuthology.orchestra.run.smithi002.stdout:Generating new minimal ceph.conf... 2024-02-16T22:12:02.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:02 smithi002 ceph-mon[28780]: from='client.? 172.21.15.2:0/668771830' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:12:03.179 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.568+0000 7f3b865fc700 1 Processor -- start 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.568+0000 7f3b865fc700 1 -- start start 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.568+0000 7f3b865fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b801077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.568+0000 7f3b865fc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b80103190 con 0x7f3b80076c00 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.568+0000 7f3b7ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b801077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.568+0000 7f3b7ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b801077a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59280/0 (socket says 172.21.15.2:59280) 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.568+0000 7f3b7ffff700 1 -- 172.21.15.2:0/559488158 learned_addr learned my addr 172.21.15.2:0/559488158 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.569+0000 7f3b7ffff700 1 -- 172.21.15.2:0/559488158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b80107ce0 con 0x7f3b80076c00 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.569+0000 7f3b7ffff700 1 --2- 172.21.15.2:0/559488158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b801077a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3b700098e0 tx=0x7f3b70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ec27fb4cb9fee3 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.569+0000 7f3b7effd700 1 -- 172.21.15.2:0/559488158 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b7000b860 con 0x7f3b80076c00 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.569+0000 7f3b7effd700 1 -- 172.21.15.2:0/559488158 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3b7000be50 con 0x7f3b80076c00 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.569+0000 7f3b7effd700 1 -- 172.21.15.2:0/559488158 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b70010380 con 0x7f3b80076c00 2024-02-16T22:12:03.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.570+0000 7f3b865fc700 1 -- 172.21.15.2:0/559488158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 msgr2=0x7f3b801077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.570+0000 7f3b865fc700 1 --2- 172.21.15.2:0/559488158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b801077a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3b700098e0 tx=0x7f3b70009250 comp rx=0 tx=0).stop 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.570+0000 7f3b865fc700 1 -- 172.21.15.2:0/559488158 shutdown_connections 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.570+0000 7f3b865fc700 1 --2- 172.21.15.2:0/559488158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b801077a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.570+0000 7f3b865fc700 1 -- 172.21.15.2:0/559488158 >> 172.21.15.2:0/559488158 conn(0x7f3b800fcdd0 msgr2=0x7f3b800ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.570+0000 7f3b865fc700 1 -- 172.21.15.2:0/559488158 shutdown_connections 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.570+0000 7f3b865fc700 1 -- 172.21.15.2:0/559488158 wait complete. 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.570+0000 7f3b865fc700 1 Processor -- start 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.570+0000 7f3b865fc700 1 -- start start 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b865fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b80113a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b865fc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b80113f70 con 0x7f3b80076c00 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b7ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b80113a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b7ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b80113a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59292/0 (socket says 172.21.15.2:59292) 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b7ffff700 1 -- 172.21.15.2:0/668771830 learned_addr learned my addr 172.21.15.2:0/668771830 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:03.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b7ffff700 1 -- 172.21.15.2:0/668771830 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b700094f0 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b7ffff700 1 --2- 172.21.15.2:0/668771830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b80113a30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000b440 tx=0x7f3b70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b7cff9700 1 -- 172.21.15.2:0/668771830 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b700107c0 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b7cff9700 1 -- 172.21.15.2:0/668771830 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3b70010db0 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b865fc700 1 -- 172.21.15.2:0/668771830 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b80114170 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b7cff9700 1 -- 172.21.15.2:0/668771830 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b7001ad30 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b865fc700 1 -- 172.21.15.2:0/668771830 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b80114e10 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.571+0000 7f3b7cff9700 1 -- 172.21.15.2:0/668771830 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3b7002e6c0 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.572+0000 7f3b7cff9700 1 -- 172.21.15.2:0/668771830 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3b700249b0 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.572+0000 7f3b865fc700 1 -- 172.21.15.2:0/668771830 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8005f5c0 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.575+0000 7f3b7cff9700 1 -- 172.21.15.2:0/668771830 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3b7001a430 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.644+0000 7f3b865fc700 1 -- 172.21.15.2:0/668771830 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3b80064ad0 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.644+0000 7f3b7cff9700 1 -- 172.21.15.2:0/668771830 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7f3b7002eba0 con 0x7f3b80076c00 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.645+0000 7f3b865fc700 1 -- 172.21.15.2:0/668771830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 msgr2=0x7f3b80113a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.645+0000 7f3b865fc700 1 --2- 172.21.15.2:0/668771830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b80113a30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000b440 tx=0x7f3b70009250 comp rx=0 tx=0).stop 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.645+0000 7f3b865fc700 1 -- 172.21.15.2:0/668771830 shutdown_connections 2024-02-16T22:12:03.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.645+0000 7f3b865fc700 1 --2- 172.21.15.2:0/668771830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b80076c00 0x7f3b80113a30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:03.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.645+0000 7f3b865fc700 1 -- 172.21.15.2:0/668771830 >> 172.21.15.2:0/668771830 conn(0x7f3b800fcdd0 msgr2=0x7f3b800fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:03.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.645+0000 7f3b865fc700 1 -- 172.21.15.2:0/668771830 shutdown_connections 2024-02-16T22:12:03.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:02.645+0000 7f3b865fc700 1 -- 172.21.15.2:0/668771830 wait complete. 2024-02-16T22:12:03.183 INFO:teuthology.orchestra.run.smithi002.stdout:Restarting the monitor... 2024-02-16T22:12:03.443 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:03 smithi002 systemd[1]: Stopping Ceph mon.smithi002 for 43c11c36-cd18-11ee-95ba-87774f69a715... 2024-02-16T22:12:03.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[28776]: 2024-02-16T22:12:03.441+0000 7f9e1c4cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi002 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-16T22:12:03.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[28776]: 2024-02-16T22:12:03.441+0000 7f9e1c4cd700 -1 mon.smithi002@0(leader) e1 *** Got Signal Terminated *** 2024-02-16T22:12:03.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:03 smithi002 podman[29024]: 2024-02-16 22:12:03.61505343 +0000 UTC m=+0.272367600 container died a0370ab274c87a3987b97ed2ea3234edd40ac2a3013ef87e7fc01d8194cd4f85 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-b88fae6, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-02-16T22:12:04.218 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:03 smithi002 podman[29024]: 2024-02-16 22:12:03.957711643 +0000 UTC m=+0.615025854 container cleanup a0370ab274c87a3987b97ed2ea3234edd40ac2a3013ef87e7fc01d8194cd4f85 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-b88fae6, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-16T22:12:04.218 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:03 smithi002 bash[29024]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002 2024-02-16T22:12:04.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 podman[29038]: 2024-02-16 22:12:04.216833403 +0000 UTC m=+0.603256529 container remove a0370ab274c87a3987b97ed2ea3234edd40ac2a3013ef87e7fc01d8194cd4f85 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=quincy-b88fae6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-16T22:12:04.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 systemd[1]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mon.smithi002.service: Deactivated successfully. 2024-02-16T22:12:04.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 systemd[1]: Stopped Ceph mon.smithi002 for 43c11c36-cd18-11ee-95ba-87774f69a715. 2024-02-16T22:12:04.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 systemd[1]: Starting Ceph mon.smithi002 for 43c11c36-cd18-11ee-95ba-87774f69a715... 2024-02-16T22:12:04.840 INFO:teuthology.orchestra.run.smithi002.stdout:Setting mon public_network to 172.21.0.0/20 2024-02-16T22:12:04.869 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 podman[29135]: 2024-02-16 22:12:04.503179484 +0000 UTC m=+0.017606159 image pull 870a0944ccb83fae8829e20e516a8aeec00e19c6a392cc9e32528216b9cb93f0 quay.ceph.io/ceph-ci/ceph:quincy 2024-02-16T22:12:04.869 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 podman[29135]: 2024-02-16 22:12:04.601184298 +0000 UTC m=+0.115610972 container create cc6b0b4bb5879d93ff1037254fc5bbb2b726169e6669a5bc06f328c3d072a727 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-b88fae6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-16T22:12:04.869 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 podman[29135]: 2024-02-16 22:12:04.76995135 +0000 UTC m=+0.284378023 container init cc6b0b4bb5879d93ff1037254fc5bbb2b726169e6669a5bc06f328c3d072a727 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-b88fae6, org.label-schema.schema-version=1.0) 2024-02-16T22:12:04.869 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 podman[29135]: 2024-02-16 22:12:04.772838959 +0000 UTC m=+0.287265641 container start cc6b0b4bb5879d93ff1037254fc5bbb2b726169e6669a5bc06f328c3d072a727 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, RELEASE=quincy-b88fae6, org.label-schema.schema-version=1.0, ceph=True) 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: set uid:gid to 167:167 (ceph:ceph) 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable), process ceph-mon, pid 2 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: pidfile_write: ignore empty --pid-file 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: load: jerasure load: lrc 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: RocksDB version: 6.15.5 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Compile date Feb 15 2024 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: DB SUMMARY 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: DB Session ID: K8L94ZT8JSCDJ69UF7L0 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: CURRENT file: CURRENT 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: IDENTITY file: IDENTITY 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi002/store.db dir, Total Num: 1, files: 000008.sst 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi002/store.db: 000010.log size: 97246 ; 2024-02-16T22:12:04.870 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.error_if_exists: 0 2024-02-16T22:12:04.871 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.create_if_missing: 0 2024-02-16T22:12:04.871 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.paranoid_checks: 1 2024-02-16T22:12:04.871 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-16T22:12:04.871 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.env: 0x55587de05a80 2024-02-16T22:12:04.871 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.fs: Posix File System 2024-02-16T22:12:04.871 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.info_log: 0x55587eb49cc0 2024-02-16T22:12:04.871 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_file_opening_threads: 16 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.statistics: (nil) 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.use_fsync: 0 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_log_file_size: 0 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.keep_log_file_num: 1000 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.recycle_log_file_num: 0 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.allow_fallocate: 1 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.allow_mmap_reads: 0 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.allow_mmap_writes: 0 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.use_direct_reads: 0 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.create_missing_column_families: 0 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.db_log_dir: 2024-02-16T22:12:04.872 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi002/store.db 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.advise_random_on_open: 1 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.db_write_buffer_size: 0 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.write_buffer_manager: 0x55587f8beb40 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-16T22:12:04.873 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.rate_limiter: (nil) 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.wal_recovery_mode: 2 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.enable_thread_tracking: 0 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.enable_pipelined_write: 0 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.unordered_write: 0 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.row_cache: None 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.wal_filter: None 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.allow_ingest_behind: 0 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.preserve_deletes: 0 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.two_write_queues: 0 2024-02-16T22:12:04.874 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.manual_wal_flush: 0 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.atomic_flush: 0 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.log_readahead_size: 0 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.best_efforts_recovery: 0 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.allow_data_in_errors: 0 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.db_host_id: __hostname__ 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_background_jobs: 2 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_background_compactions: -1 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_subcompactions: 1 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-16T22:12:04.875 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_total_wal_size: 0 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_open_files: -1 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bytes_per_sync: 0 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_readahead_size: 0 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_background_flushes: -1 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Compression algorithms supported: 2024-02-16T22:12:04.876 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: kZSTD supported: 0 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: kXpressCompression supported: 0 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: kLZ4HCCompression supported: 1 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: kZlibCompression supported: 1 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: kSnappyCompression supported: 1 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: kLZ4Compression supported: 1 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: kBZip2Compression supported: 0 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi002/store.db/MANIFEST-000009 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.merge_operator: 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_filter: None 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_filter_factory: None 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.sst_partitioner_factory: None 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-16T22:12:04.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55587eb9ef58) 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: index_type: 0 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: data_block_index_type: 0 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: index_shortening: 1 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: hash_index_allow_collision: 1 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: checksum: 1 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: no_block_cache: 0 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: block_cache: 0x55587ebd1090 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: block_cache_options: 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: capacity : 536870912 2024-02-16T22:12:04.878 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: num_shard_bits : 4 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: strict_capacity_limit : 0 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: block_cache_compressed: (nil) 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: persistent_cache: (nil) 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: block_size: 4096 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: block_size_deviation: 10 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: block_restart_interval: 16 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: index_block_restart_interval: 1 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: metadata_block_size: 4096 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: partition_filters: 0 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: use_delta_encoding: 1 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-16T22:12:04.879 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: whole_key_filtering: 1 2024-02-16T22:12:04.880 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: verify_compression: 0 2024-02-16T22:12:04.880 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-02-16T22:12:04.880 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: format_version: 4 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: enable_index_compression: 1 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: block_align: 0 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.write_buffer_size: 33554432 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_write_buffer_number: 2 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compression: NoCompression 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bottommost_compression: Disabled 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.prefix_extractor: nullptr 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.num_levels: 7 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-16T22:12:04.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compression_opts.level: 32767 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compression_opts.strategy: 0 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compression_opts.enabled: false 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.target_file_size_base: 67108864 2024-02-16T22:12:04.882 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.arena_block_size: 4194304 2024-02-16T22:12:04.883 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.disable_auto_compactions: 0 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-16T22:12:04.884 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.table_properties_collectors: 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.inplace_update_support: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.bloom_locality: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.max_successive_merges: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.paranoid_file_checks: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.force_consistency_checks: 1 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.report_bg_io_stats: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.ttl: 2592000 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.enable_blob_files: false 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.min_blob_size: 0 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.blob_file_size: 268435456 2024-02-16T22:12:04.885 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi002/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708121524797863, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708121524798670, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92518, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90613, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83328, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708121524, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "b7c56271-ffa5-4d9e-a7cf-91fe74de8311", "db_session_id": "K8L94ZT8JSCDJ69UF7L0"}} 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708121524799007, "job": 1, "event": "recovery_finished"} 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi002/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55587eb65180 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: DB pointer 0x55587ec3a000 2024-02-16T22:12:04.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ** DB Stats ** 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: L0 2/0 92.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.1 0.00 0.00 1 0.001 0 0 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Sum 2/0 92.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.1 0.00 0.00 1 0.001 0 0 2024-02-16T22:12:04.887 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.1 0.00 0.00 1 0.001 0 0 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.1 0.00 0.00 1 0.001 0 0 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Cumulative compaction: 0.00 GB write, 29.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Interval compaction: 0.00 GB write, 29.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-16T22:12:04.888 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: L0 2/0 92.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.1 0.00 0.00 1 0.001 0 0 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Sum 2/0 92.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.1 0.00 0.00 1 0.001 0 0 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ** Compaction Stats [default] ** 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.1 0.00 0.00 1 0.001 0 0 2024-02-16T22:12:04.889 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Cumulative compaction: 0.00 GB write, 29.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: starting mon.smithi002 rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi002 fsid 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mon.smithi002@-1(???) e1 preinit fsid 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mon.smithi002@-1(???).mds e1 new map 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mon.smithi002@-1(???).mds e1 print_map 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: e1 2024-02-16T22:12:04.890 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: legacy client fscid: -1 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout: No filesystems configured 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mon.smithi002@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mon.smithi002@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mon.smithi002@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mon.smithi002@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mon.smithi002@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 bash[29135]: cc6b0b4bb5879d93ff1037254fc5bbb2b726169e6669a5bc06f328c3d072a727 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 systemd[1]: Started Ceph mon.smithi002 for 43c11c36-cd18-11ee-95ba-87774f69a715. 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mon.smithi002 is new leader, mons smithi002 in quorum (ranks 0) 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: monmap e1: 1 mons at {smithi002=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: fsmap 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: osdmap e1: 0 total, 0 up, 0 in 2024-02-16T22:12:04.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:04 smithi002 ceph-mon[29154]: mgrmap e1: no daemons active 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.356+0000 7f09b5ba2700 1 Processor -- start 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.356+0000 7f09b5ba2700 1 -- start start 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.357+0000 7f09b5ba2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.357+0000 7f09b5ba2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b01030c0 con 0x7f09b0102770 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.357+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.357+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59294/0 (socket says 172.21.15.2:59294) 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.357+0000 7f09af7fe700 1 -- 172.21.15.2:0/363773680 learned_addr learned my addr 172.21.15.2:0/363773680 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.357+0000 7f09af7fe700 1 -- 172.21.15.2:0/363773680 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b0103200 con 0x7f09b0102770 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.357+0000 7f09af7fe700 1 --2- 172.21.15.2:0/363773680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0102b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f09a00098e0 tx=0x7f09a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5ed4ace4416428 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09ae7fc700 1 -- 172.21.15.2:0/363773680 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a000b860 con 0x7f09b0102770 2024-02-16T22:12:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09ae7fc700 1 -- 172.21.15.2:0/363773680 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f09a000be50 con 0x7f09b0102770 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09ae7fc700 1 -- 172.21.15.2:0/363773680 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a0010380 con 0x7f09b0102770 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09b5ba2700 1 -- 172.21.15.2:0/363773680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 msgr2=0x7f09b0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09b5ba2700 1 --2- 172.21.15.2:0/363773680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0102b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f09a00098e0 tx=0x7f09a0009250 comp rx=0 tx=0).stop 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09b5ba2700 1 -- 172.21.15.2:0/363773680 shutdown_connections 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09b5ba2700 1 --2- 172.21.15.2:0/363773680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0102b80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09b5ba2700 1 -- 172.21.15.2:0/363773680 >> 172.21.15.2:0/363773680 conn(0x7f09b00fded0 msgr2=0x7f09b0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09b5ba2700 1 -- 172.21.15.2:0/363773680 shutdown_connections 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.358+0000 7f09b5ba2700 1 -- 172.21.15.2:0/363773680 wait complete. 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.359+0000 7f09b5ba2700 1 Processor -- start 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.359+0000 7f09b5ba2700 1 -- start start 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.359+0000 7f09b5ba2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.359+0000 7f09b5ba2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0111ad0 con 0x7f09b0102770 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.359+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:05.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.359+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59306/0 (socket says 172.21.15.2:59306) 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.359+0000 7f09af7fe700 1 -- 172.21.15.2:0/1591564007 learned_addr learned my addr 172.21.15.2:0/1591564007 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.359+0000 7f09af7fe700 1 -- 172.21.15.2:0/1591564007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a00094f0 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.360+0000 7f09af7fe700 1 --2- 172.21.15.2:0/1591564007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0111590 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f09a000b440 tx=0x7f09a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.360+0000 7f0997fff700 1 -- 172.21.15.2:0/1591564007 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a0010760 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.360+0000 7f09b5ba2700 1 -- 172.21.15.2:0/1591564007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b0111cd0 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.360+0000 7f0997fff700 1 -- 172.21.15.2:0/1591564007 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f09a0010d50 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.360+0000 7f0997fff700 1 -- 172.21.15.2:0/1591564007 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a001ae20 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.360+0000 7f09b5ba2700 1 -- 172.21.15.2:0/1591564007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b0112970 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.360+0000 7f0997fff700 1 -- 172.21.15.2:0/1591564007 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f09a002e6c0 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.360+0000 7f0997fff700 1 -- 172.21.15.2:0/1591564007 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f09a0024900 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.361+0000 7f09b5ba2700 1 -- 172.21.15.2:0/1591564007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b005f5c0 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.363+0000 7f0997fff700 1 -- 172.21.15.2:0/1591564007 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f09a0029070 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.432+0000 7f09b5ba2700 1 -- 172.21.15.2:0/1591564007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f09b01045e0 con 0x7f09b0102770 2024-02-16T22:12:05.962 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.434+0000 7f0997fff700 1 -- 172.21.15.2:0/1591564007 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f09a001a6a0 con 0x7f09b0102770 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.435+0000 7f09b5ba2700 1 -- 172.21.15.2:0/1591564007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 msgr2=0x7f09b0111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.435+0000 7f09b5ba2700 1 --2- 172.21.15.2:0/1591564007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0111590 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f09a000b440 tx=0x7f09a0009250 comp rx=0 tx=0).stop 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.435+0000 7f09b5ba2700 1 -- 172.21.15.2:0/1591564007 shutdown_connections 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.435+0000 7f09b5ba2700 1 --2- 172.21.15.2:0/1591564007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09b0102770 0x7f09b0111590 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.435+0000 7f09b5ba2700 1 -- 172.21.15.2:0/1591564007 >> 172.21.15.2:0/1591564007 conn(0x7f09b00fded0 msgr2=0x7f09b00ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.435+0000 7f09b5ba2700 1 -- 172.21.15.2:0/1591564007 shutdown_connections 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:05.435+0000 7f09b5ba2700 1 -- 172.21.15.2:0/1591564007 wait complete. 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mgr... 2024-02-16T22:12:05.963 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 9283 ... 2024-02-16T22:12:06.136 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mgr.smithi002.resmjv 2024-02-16T22:12:06.136 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mgr.smithi002.resmjv.service: Unit ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mgr.smithi002.resmjv.service not loaded. 2024-02-16T22:12:06.283 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-43c11c36-cd18-11ee-95ba-87774f69a715.target.wants/ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mgr.smithi002.resmjv.service → /etc/systemd/system/ceph-43c11c36-cd18-11ee-95ba-87774f69a715@.service. 2024-02-16T22:12:06.706 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:06 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1591564007' entity='client.admin' 2024-02-16T22:12:06.820 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-16T22:12:06.820 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-02-16T22:12:06.828 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-16T22:12:06.828 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-02-16T22:12:06.828 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-02-16T22:12:06.828 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2024-02-16T22:12:06.836 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-16T22:12:06.836 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-02-16T22:12:06.844 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-16T22:12:06.844 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-02-16T22:12:06.844 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-02-16T22:12:06.844 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-16T22:12:06.844 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr to start... 2024-02-16T22:12:06.844 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr... 2024-02-16T22:12:08.163 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:07 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/827239412' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-16T22:12:08.543 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:08.543 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-02-16T22:12:08.543 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "43c11c36-cd18-11ee-95ba-87774f69a715", 2024-02-16T22:12:08.543 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-02-16T22:12:08.543 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-16T22:12:08.543 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-16T22:12:08.543 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-16T22:12:08.543 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "smithi002" 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:08.544 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-16T22:12:08.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:08.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-16T22:12:08.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-16T22:12:08.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-16T22:12:08.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-16T22:12:08.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-16T22:12:08.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-16T22:12:08.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:08.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-16T22:12:08.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-16T22:12:08.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-16T22:12:08.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-02-16T22:11:59.287049+0000", 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.798+0000 7fa70b0c8700 1 Processor -- start 2024-02-16T22:12:08.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.798+0000 7fa70b0c8700 1 -- start start 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.798+0000 7fa70b0c8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.798+0000 7fa70b0c8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa704105960 con 0x7fa704076b80 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.798+0000 7fa708e64700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.798+0000 7fa708e64700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59338/0 (socket says 172.21.15.2:59338) 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.798+0000 7fa708e64700 1 -- 172.21.15.2:0/3214522114 learned_addr learned my addr 172.21.15.2:0/3214522114 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.798+0000 7fa708e64700 1 -- 172.21.15.2:0/3214522114 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa704105aa0 con 0x7fa704076b80 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.799+0000 7fa708e64700 1 --2- 172.21.15.2:0/3214522114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704105420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80098e0 tx=0x7fa6f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612570cd3d8d2e26 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.799+0000 7fa7037fe700 1 -- 172.21.15.2:0/3214522114 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6f800b860 con 0x7fa704076b80 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.799+0000 7fa7037fe700 1 -- 172.21.15.2:0/3214522114 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa6f800be50 con 0x7fa704076b80 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.799+0000 7fa7037fe700 1 -- 172.21.15.2:0/3214522114 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6f8010380 con 0x7fa704076b80 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.800+0000 7fa70b0c8700 1 -- 172.21.15.2:0/3214522114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 msgr2=0x7fa704105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.800+0000 7fa70b0c8700 1 --2- 172.21.15.2:0/3214522114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704105420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80098e0 tx=0x7fa6f8009250 comp rx=0 tx=0).stop 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.800+0000 7fa70b0c8700 1 -- 172.21.15.2:0/3214522114 shutdown_connections 2024-02-16T22:12:08.555 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.800+0000 7fa70b0c8700 1 --2- 172.21.15.2:0/3214522114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704105420 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.800+0000 7fa70b0c8700 1 -- 172.21.15.2:0/3214522114 >> 172.21.15.2:0/3214522114 conn(0x7fa7040fcdd0 msgr2=0x7fa7040ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.800+0000 7fa70b0c8700 1 -- 172.21.15.2:0/3214522114 shutdown_connections 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.800+0000 7fa70b0c8700 1 -- 172.21.15.2:0/3214522114 wait complete. 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.800+0000 7fa70b0c8700 1 Processor -- start 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.800+0000 7fa70b0c8700 1 -- start start 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa70b0c8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa70b0c8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa704111a80 con 0x7fa704076b80 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa708e64700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa708e64700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59348/0 (socket says 172.21.15.2:59348) 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa708e64700 1 -- 172.21.15.2:0/827239412 learned_addr learned my addr 172.21.15.2:0/827239412 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa708e64700 1 -- 172.21.15.2:0/827239412 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f80094f0 con 0x7fa704076b80 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa708e64700 1 --2- 172.21.15.2:0/827239412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704111540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800b440 tx=0x7fa6f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa7017fa700 1 -- 172.21.15.2:0/827239412 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6f80107c0 con 0x7fa704076b80 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa7017fa700 1 -- 172.21.15.2:0/827239412 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa6f8010db0 con 0x7fa704076b80 2024-02-16T22:12:08.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa70b0c8700 1 -- 172.21.15.2:0/827239412 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa704111c80 con 0x7fa704076b80 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa7017fa700 1 -- 172.21.15.2:0/827239412 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6f801ad30 con 0x7fa704076b80 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.801+0000 7fa70b0c8700 1 -- 172.21.15.2:0/827239412 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa704112920 con 0x7fa704076b80 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.802+0000 7fa7017fa700 1 -- 172.21.15.2:0/827239412 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa6f802e6c0 con 0x7fa704076b80 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.802+0000 7fa7017fa700 1 -- 172.21.15.2:0/827239412 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa6f8024950 con 0x7fa704076b80 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.802+0000 7fa70b0c8700 1 -- 172.21.15.2:0/827239412 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70405f5c0 con 0x7fa704076b80 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.804+0000 7fa7017fa700 1 -- 172.21.15.2:0/827239412 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa6f801a430 con 0x7fa704076b80 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.877+0000 7fa70b0c8700 1 -- 172.21.15.2:0/827239412 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa704106820 con 0x7fa704076b80 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.877+0000 7fa7017fa700 1 -- 172.21.15.2:0/827239412 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fa6f8010920 con 0x7fa704076b80 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.878+0000 7fa70b0c8700 1 -- 172.21.15.2:0/827239412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 msgr2=0x7fa704111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.878+0000 7fa70b0c8700 1 --2- 172.21.15.2:0/827239412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704111540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800b440 tx=0x7fa6f8009250 comp rx=0 tx=0).stop 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.879+0000 7fa70b0c8700 1 -- 172.21.15.2:0/827239412 shutdown_connections 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.879+0000 7fa70b0c8700 1 --2- 172.21.15.2:0/827239412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa704076b80 0x7fa704111540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.879+0000 7fa70b0c8700 1 -- 172.21.15.2:0/827239412 >> 172.21.15.2:0/827239412 conn(0x7fa7040fcdd0 msgr2=0x7fa7040fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.879+0000 7fa70b0c8700 1 -- 172.21.15.2:0/827239412 shutdown_connections 2024-02-16T22:12:08.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:07.879+0000 7fa70b0c8700 1 -- 172.21.15.2:0/827239412 wait complete. 2024-02-16T22:12:08.558 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (1/15)... 2024-02-16T22:12:11.350 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:11 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1352071455' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-16T22:12:11.658 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:11.658 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-02-16T22:12:11.658 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "43c11c36-cd18-11ee-95ba-87774f69a715", 2024-02-16T22:12:11.658 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-02-16T22:12:11.658 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "smithi002" 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-16T22:12:11.659 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-16T22:12:11.660 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:11.660 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-16T22:12:11.660 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:11.660 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-16T22:12:11.660 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-16T22:12:11.660 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-16T22:12:11.667 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-16T22:12:11.667 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-16T22:12:11.667 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-16T22:12:11.667 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-16T22:12:11.668 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-02-16T22:11:59.287049+0000", 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-16T22:12:11.669 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.953+0000 7f5cb51ac700 1 Processor -- start 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.954+0000 7f5cb51ac700 1 -- start start 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.954+0000 7f5cb51ac700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb00771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.954+0000 7f5cb51ac700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb00790f0 con 0x7f5cb0078ce0 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.954+0000 7f5caed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb00771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.954+0000 7f5caed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb00771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60868/0 (socket says 172.21.15.2:60868) 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.954+0000 7f5caed9d700 1 -- 172.21.15.2:0/3252087216 learned_addr learned my addr 172.21.15.2:0/3252087216 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.954+0000 7f5caed9d700 1 -- 172.21.15.2:0/3252087216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0077730 con 0x7f5cb0078ce0 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.955+0000 7f5caed9d700 1 --2- 172.21.15.2:0/3252087216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb00771f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ca40098e0 tx=0x7f5ca4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f7770d3376619d server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.955+0000 7f5cadd9b700 1 -- 172.21.15.2:0/3252087216 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca400b860 con 0x7f5cb0078ce0 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.955+0000 7f5cadd9b700 1 -- 172.21.15.2:0/3252087216 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5ca400be50 con 0x7f5cb0078ce0 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.955+0000 7f5cadd9b700 1 -- 172.21.15.2:0/3252087216 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca4010380 con 0x7f5cb0078ce0 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.955+0000 7f5cb51ac700 1 -- 172.21.15.2:0/3252087216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 msgr2=0x7f5cb00771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.955+0000 7f5cb51ac700 1 --2- 172.21.15.2:0/3252087216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb00771f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ca40098e0 tx=0x7f5ca4009250 comp rx=0 tx=0).stop 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.955+0000 7f5cb51ac700 1 -- 172.21.15.2:0/3252087216 shutdown_connections 2024-02-16T22:12:11.670 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.955+0000 7f5cb51ac700 1 --2- 172.21.15.2:0/3252087216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb00771f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.955+0000 7f5cb51ac700 1 -- 172.21.15.2:0/3252087216 >> 172.21.15.2:0/3252087216 conn(0x7f5cb00fded0 msgr2=0x7f5cb0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.956+0000 7f5cb51ac700 1 -- 172.21.15.2:0/3252087216 shutdown_connections 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.956+0000 7f5cb51ac700 1 -- 172.21.15.2:0/3252087216 wait complete. 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.956+0000 7f5cb51ac700 1 Processor -- start 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.956+0000 7f5cb51ac700 1 -- start start 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.956+0000 7f5cb51ac700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb01137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.956+0000 7f5cb51ac700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb0113d20 con 0x7f5cb0078ce0 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5caed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb01137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5caed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb01137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60872/0 (socket says 172.21.15.2:60872) 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5caed9d700 1 -- 172.21.15.2:0/1352071455 learned_addr learned my addr 172.21.15.2:0/1352071455 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5caed9d700 1 -- 172.21.15.2:0/1352071455 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca40094f0 con 0x7f5cb0078ce0 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5caed9d700 1 --2- 172.21.15.2:0/1352071455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb01137e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400b440 tx=0x7f5ca4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5c977fe700 1 -- 172.21.15.2:0/1352071455 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca4010760 con 0x7f5cb0078ce0 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5c977fe700 1 -- 172.21.15.2:0/1352071455 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5ca4010d50 con 0x7f5cb0078ce0 2024-02-16T22:12:11.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5cb51ac700 1 -- 172.21.15.2:0/1352071455 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb0113f20 con 0x7f5cb0078ce0 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5c977fe700 1 -- 172.21.15.2:0/1352071455 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca401adc0 con 0x7f5cb0078ce0 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5cb51ac700 1 -- 172.21.15.2:0/1352071455 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb0114bc0 con 0x7f5cb0078ce0 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5c977fe700 1 -- 172.21.15.2:0/1352071455 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5ca4019690 con 0x7f5cb0078ce0 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.957+0000 7f5c977fe700 1 -- 172.21.15.2:0/1352071455 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5ca4024900 con 0x7f5cb0078ce0 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.958+0000 7f5cb51ac700 1 -- 172.21.15.2:0/1352071455 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb005f5c0 con 0x7f5cb0078ce0 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:10.960+0000 7f5c977fe700 1 -- 172.21.15.2:0/1352071455 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5ca4029060 con 0x7f5cb0078ce0 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:11.033+0000 7f5cb51ac700 1 -- 172.21.15.2:0/1352071455 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5cb0108a40 con 0x7f5cb0078ce0 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:11.033+0000 7f5c977fe700 1 -- 172.21.15.2:0/1352071455 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f5ca401a5f0 con 0x7f5cb0078ce0 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:11.035+0000 7f5cb51ac700 1 -- 172.21.15.2:0/1352071455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 msgr2=0x7f5cb01137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:11.035+0000 7f5cb51ac700 1 --2- 172.21.15.2:0/1352071455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb01137e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400b440 tx=0x7f5ca4009250 comp rx=0 tx=0).stop 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:11.035+0000 7f5cb51ac700 1 -- 172.21.15.2:0/1352071455 shutdown_connections 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:11.035+0000 7f5cb51ac700 1 --2- 172.21.15.2:0/1352071455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cb0078ce0 0x7f5cb01137e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:11.035+0000 7f5cb51ac700 1 -- 172.21.15.2:0/1352071455 >> 172.21.15.2:0/1352071455 conn(0x7f5cb00fded0 msgr2=0x7f5cb00ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:11.035+0000 7f5cb51ac700 1 -- 172.21.15.2:0/1352071455 shutdown_connections 2024-02-16T22:12:11.672 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:11.035+0000 7f5cb51ac700 1 -- 172.21.15.2:0/1352071455 wait complete. 2024-02-16T22:12:11.673 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (2/15)... 2024-02-16T22:12:13.785 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: Activating manager daemon smithi002.resmjv 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: mgrmap e2: smithi002.resmjv(active, starting, since 0.00270498s) 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr metadata", "who": "smithi002.resmjv", "id": "smithi002.resmjv"}]: dispatch 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: Manager daemon smithi002.resmjv is now available 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/mirror_snapshot_schedule"}]: dispatch 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/trash_purge_schedule"}]: dispatch 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:13.786 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:13 smithi002 ceph-mon[29154]: from='mgr.14100 172.21.15.2:0/803160589' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:14.789 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:14 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/847120263' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-16T22:12:14.789 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:14 smithi002 ceph-mon[29154]: mgrmap e3: smithi002.resmjv(active, since 1.0056s) 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "43c11c36-cd18-11ee-95ba-87774f69a715", 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-16T22:12:14.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "smithi002" 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-16T22:12:14.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-16T22:12:14.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-02-16T22:11:59.287049+0000", 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-16T22:12:14.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.081+0000 7f435c099700 1 Processor -- start 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.082+0000 7f435c099700 1 -- start start 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.082+0000 7f435c099700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354104250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.082+0000 7f435c099700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4354104790 con 0x7f4354103e40 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.082+0000 7f4359e35700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354104250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.082+0000 7f4359e35700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354104250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60926/0 (socket says 172.21.15.2:60926) 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.082+0000 7f4359e35700 1 -- 172.21.15.2:0/3749491435 learned_addr learned my addr 172.21.15.2:0/3749491435 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.082+0000 7f4359e35700 1 -- 172.21.15.2:0/3749491435 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43541048d0 con 0x7f4354103e40 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.082+0000 7f4359e35700 1 --2- 172.21.15.2:0/3749491435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354104250 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f43440098e0 tx=0x7f4344009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8ce347990952d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f4358e33700 1 -- 172.21.15.2:0/3749491435 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f434400b860 con 0x7f4354103e40 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f4358e33700 1 -- 172.21.15.2:0/3749491435 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f434400be50 con 0x7f4354103e40 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f4358e33700 1 -- 172.21.15.2:0/3749491435 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4344010380 con 0x7f4354103e40 2024-02-16T22:12:14.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f435c099700 1 -- 172.21.15.2:0/3749491435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 msgr2=0x7f4354104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f435c099700 1 --2- 172.21.15.2:0/3749491435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354104250 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f43440098e0 tx=0x7f4344009250 comp rx=0 tx=0).stop 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f435c099700 1 -- 172.21.15.2:0/3749491435 shutdown_connections 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f435c099700 1 --2- 172.21.15.2:0/3749491435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354104250 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f435c099700 1 -- 172.21.15.2:0/3749491435 >> 172.21.15.2:0/3749491435 conn(0x7f43540fc580 msgr2=0x7f43540fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f435c099700 1 -- 172.21.15.2:0/3749491435 shutdown_connections 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.083+0000 7f435c099700 1 -- 172.21.15.2:0/3749491435 wait complete. 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.084+0000 7f435c099700 1 Processor -- start 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.084+0000 7f435c099700 1 -- start start 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.084+0000 7f435c099700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.084+0000 7f435c099700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4354111aa0 con 0x7f4354103e40 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.084+0000 7f4359e35700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.084+0000 7f4359e35700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60928/0 (socket says 172.21.15.2:60928) 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.084+0000 7f4359e35700 1 -- 172.21.15.2:0/847120263 learned_addr learned my addr 172.21.15.2:0/847120263 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:14.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.084+0000 7f4359e35700 1 -- 172.21.15.2:0/847120263 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43440094f0 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.084+0000 7f4359e35700 1 --2- 172.21.15.2:0/847120263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354111560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4344000c00 tx=0x7f4344009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.085+0000 7f434a7fc700 1 -- 172.21.15.2:0/847120263 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43440107c0 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.085+0000 7f434a7fc700 1 -- 172.21.15.2:0/847120263 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4344010db0 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.085+0000 7f435c099700 1 -- 172.21.15.2:0/847120263 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4354111ca0 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.085+0000 7f434a7fc700 1 -- 172.21.15.2:0/847120263 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f434401ad00 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.085+0000 7f435c099700 1 -- 172.21.15.2:0/847120263 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4354112940 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.085+0000 7f434a7fc700 1 -- 172.21.15.2:0/847120263 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7f43440193c0 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.086+0000 7f434a7fc700 1 -- 172.21.15.2:0/847120263 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f434404a1e0 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.086+0000 7f435c099700 1 -- 172.21.15.2:0/847120263 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4354112130 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.088+0000 7f434a7fc700 1 -- 172.21.15.2:0/847120263 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4344022930 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.159+0000 7f435c099700 1 -- 172.21.15.2:0/847120263 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4354112b80 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.160+0000 7f434a7fc700 1 -- 172.21.15.2:0/847120263 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f4344022340 con 0x7f4354103e40 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.161+0000 7f435c099700 1 -- 172.21.15.2:0/847120263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 msgr2=0x7f4354111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.161+0000 7f435c099700 1 --2- 172.21.15.2:0/847120263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354111560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4344000c00 tx=0x7f4344009250 comp rx=0 tx=0).stop 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.161+0000 7f435c099700 1 -- 172.21.15.2:0/847120263 shutdown_connections 2024-02-16T22:12:14.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.161+0000 7f435c099700 1 --2- 172.21.15.2:0/847120263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4354103e40 0x7f4354111560 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:14.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.161+0000 7f435c099700 1 -- 172.21.15.2:0/847120263 >> 172.21.15.2:0/847120263 conn(0x7f43540fc580 msgr2=0x7f43540fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:14.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.161+0000 7f435c099700 1 -- 172.21.15.2:0/847120263 shutdown_connections 2024-02-16T22:12:14.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:14.161+0000 7f435c099700 1 -- 172.21.15.2:0/847120263 wait complete. 2024-02-16T22:12:14.811 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (3/15)... 2024-02-16T22:12:17.120 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:16 smithi002 ceph-mon[29154]: mgrmap e4: smithi002.resmjv(active, since 2s) 2024-02-16T22:12:18.148 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:17 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1358319168' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-16T22:12:18.153 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:18.153 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-02-16T22:12:18.153 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "43c11c36-cd18-11ee-95ba-87774f69a715", 2024-02-16T22:12:18.153 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-02-16T22:12:18.153 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-16T22:12:18.153 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-16T22:12:18.153 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "smithi002" 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-16T22:12:18.155 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:18.155 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-16T22:12:18.155 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-16T22:12:18.164 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-16T22:12:18.165 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-02-16T22:11:59.287049+0000", 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.221+0000 7fb20938c700 1 Processor -- start 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.221+0000 7fb20938c700 1 -- start start 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.221+0000 7fb20938c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.221+0000 7fb20938c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2041030c0 con 0x7fb204102770 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.221+0000 7fb202ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:18.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.221+0000 7fb202ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60936/0 (socket says 172.21.15.2:60936) 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.221+0000 7fb202ffd700 1 -- 172.21.15.2:0/4251048748 learned_addr learned my addr 172.21.15.2:0/4251048748 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.221+0000 7fb202ffd700 1 -- 172.21.15.2:0/4251048748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb204103200 con 0x7fb204102770 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb202ffd700 1 --2- 172.21.15.2:0/4251048748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204102b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40098e0 tx=0x7fb1f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29eb44878217acc server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb201ffb700 1 -- 172.21.15.2:0/4251048748 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f400b860 con 0x7fb204102770 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb201ffb700 1 -- 172.21.15.2:0/4251048748 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb1f400be50 con 0x7fb204102770 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb201ffb700 1 -- 172.21.15.2:0/4251048748 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f4010380 con 0x7fb204102770 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb20938c700 1 -- 172.21.15.2:0/4251048748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 msgr2=0x7fb204102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb20938c700 1 --2- 172.21.15.2:0/4251048748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204102b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40098e0 tx=0x7fb1f4009250 comp rx=0 tx=0).stop 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb20938c700 1 -- 172.21.15.2:0/4251048748 shutdown_connections 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb20938c700 1 --2- 172.21.15.2:0/4251048748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204102b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb20938c700 1 -- 172.21.15.2:0/4251048748 >> 172.21.15.2:0/4251048748 conn(0x7fb2040fded0 msgr2=0x7fb204100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb20938c700 1 -- 172.21.15.2:0/4251048748 shutdown_connections 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.222+0000 7fb20938c700 1 -- 172.21.15.2:0/4251048748 wait complete. 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.223+0000 7fb20938c700 1 Processor -- start 2024-02-16T22:12:18.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.223+0000 7fb20938c700 1 -- start start 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.223+0000 7fb20938c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.223+0000 7fb20938c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb204077190 con 0x7fb204102770 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.223+0000 7fb202ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.223+0000 7fb202ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60944/0 (socket says 172.21.15.2:60944) 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.223+0000 7fb202ffd700 1 -- 172.21.15.2:0/1358319168 learned_addr learned my addr 172.21.15.2:0/1358319168 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.224+0000 7fb202ffd700 1 -- 172.21.15.2:0/1358319168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f40094f0 con 0x7fb204102770 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.224+0000 7fb202ffd700 1 --2- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204078d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4000c00 tx=0x7fb1f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.224+0000 7fb1eb7fe700 1 -- 172.21.15.2:0/1358319168 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f40107c0 con 0x7fb204102770 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.224+0000 7fb1eb7fe700 1 -- 172.21.15.2:0/1358319168 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb1f4010db0 con 0x7fb204102770 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.224+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb204077390 con 0x7fb204102770 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.224+0000 7fb1eb7fe700 1 -- 172.21.15.2:0/1358319168 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f401ad00 con 0x7fb204102770 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.224+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb204077820 con 0x7fb204102770 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.225+0000 7fb1eb7fe700 1 -- 172.21.15.2:0/1358319168 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fb1f401a530 con 0x7fb204102770 2024-02-16T22:12:18.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.225+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20404bed0 con 0x7fb204102770 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.225+0000 7fb1eb7fe700 1 --2- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fb1ec036740 0x7fb1ec038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.225+0000 7fb1eb7fe700 1 -- 172.21.15.2:0/1358319168 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb1f404a600 con 0x7fb204102770 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.225+0000 7fb2027fc700 1 --2- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fb1ec036740 0x7fb1ec038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.226+0000 7fb2027fc700 1 --2- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fb1ec036740 0x7fb1ec038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009990 tx=0x7fb1f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.230+0000 7fb1eb7fe700 1 -- 172.21.15.2:0/1358319168 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb1f4010920 con 0x7fb204102770 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.499+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb204077fa0 con 0x7fb204102770 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.499+0000 7fb1eb7fe700 1 -- 172.21.15.2:0/1358319168 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fb1f40233a0 con 0x7fb204102770 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fb1ec036740 msgr2=0x7fb1ec038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 --2- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fb1ec036740 0x7fb1ec038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009990 tx=0x7fb1f8006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 msgr2=0x7fb204078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 --2- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204078d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4000c00 tx=0x7fb1f4009250 comp rx=0 tx=0).stop 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 shutdown_connections 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 --2- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fb1ec036740 0x7fb1ec038bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 --2- 172.21.15.2:0/1358319168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb204102770 0x7fb204078d10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:18.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 >> 172.21.15.2:0/1358319168 conn(0x7fb2040fded0 msgr2=0x7fb2040ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:18.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 shutdown_connections 2024-02-16T22:12:18.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:17.501+0000 7fb20938c700 1 -- 172.21.15.2:0/1358319168 wait complete. 2024-02-16T22:12:18.170 INFO:teuthology.orchestra.run.smithi002.stdout:mgr is available 2024-02-16T22:12:19.238 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:18 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2314554643' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-16T22:12:19.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:19.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2024-02-16T22:12:19.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:12:19.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-16T22:12:19.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-16T22:12:19.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-16T22:12:19.497 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.729+0000 7f605cc09700 1 Processor -- start 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.729+0000 7f605cc09700 1 -- start start 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.729+0000 7f605cc09700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.729+0000 7f605cc09700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60581030c0 con 0x7f6058102770 2024-02-16T22:12:19.498 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.730+0000 7f605659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.730+0000 7f605659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60956/0 (socket says 172.21.15.2:60956) 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.730+0000 7f605659c700 1 -- 172.21.15.2:0/1959327486 learned_addr learned my addr 172.21.15.2:0/1959327486 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.730+0000 7f605659c700 1 -- 172.21.15.2:0/1959327486 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6058103200 con 0x7f6058102770 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.730+0000 7f605659c700 1 --2- 172.21.15.2:0/1959327486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6050009e50 tx=0x7f6050009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878350cdde50da72 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.730+0000 7f605559a700 1 -- 172.21.15.2:0/1959327486 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60500092e0 con 0x7f6058102770 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.730+0000 7f605559a700 1 -- 172.21.15.2:0/1959327486 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6050012430 con 0x7f6058102770 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.730+0000 7f605559a700 1 -- 172.21.15.2:0/1959327486 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60500075c0 con 0x7f6058102770 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.731+0000 7f605cc09700 1 -- 172.21.15.2:0/1959327486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 msgr2=0x7f6058102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.731+0000 7f605cc09700 1 --2- 172.21.15.2:0/1959327486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6050009e50 tx=0x7f6050009840 comp rx=0 tx=0).stop 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.731+0000 7f605cc09700 1 -- 172.21.15.2:0/1959327486 shutdown_connections 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.731+0000 7f605cc09700 1 --2- 172.21.15.2:0/1959327486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058102b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.731+0000 7f605cc09700 1 -- 172.21.15.2:0/1959327486 >> 172.21.15.2:0/1959327486 conn(0x7f60580fded0 msgr2=0x7f6058100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.731+0000 7f605cc09700 1 -- 172.21.15.2:0/1959327486 shutdown_connections 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.731+0000 7f605cc09700 1 -- 172.21.15.2:0/1959327486 wait complete. 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.731+0000 7f605cc09700 1 Processor -- start 2024-02-16T22:12:19.499 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.731+0000 7f605cc09700 1 -- start start 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f605cc09700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058111530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f605cc09700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6058111a70 con 0x7f6058102770 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f605659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058111530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f605659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058111530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60966/0 (socket says 172.21.15.2:60966) 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f605659c700 1 -- 172.21.15.2:0/2314554643 learned_addr learned my addr 172.21.15.2:0/2314554643 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f605659c700 1 -- 172.21.15.2:0/2314554643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6050009ae0 con 0x7f6058102770 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f605659c700 1 --2- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058111530 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f60500127d0 tx=0x7f6050009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f604effd700 1 -- 172.21.15.2:0/2314554643 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f605000bc30 con 0x7f6058102770 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f604effd700 1 -- 172.21.15.2:0/2314554643 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f605000bd90 con 0x7f6058102770 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6058111c70 con 0x7f6058102770 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f604effd700 1 -- 172.21.15.2:0/2314554643 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f605001c9d0 con 0x7f6058102770 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.732+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6058112910 con 0x7f6058102770 2024-02-16T22:12:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.733+0000 7f604effd700 1 -- 172.21.15.2:0/2314554643 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f6050019070 con 0x7f6058102770 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.733+0000 7f604effd700 1 --2- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f6044036790 0x7f6044038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.733+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605805f5c0 con 0x7f6058102770 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.734+0000 7f604effd700 1 -- 172.21.15.2:0/2314554643 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6050049ff0 con 0x7f6058102770 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.734+0000 7f6055d9b700 1 --2- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f6044036790 0x7f6044038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.734+0000 7f6055d9b700 1 --2- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f6044036790 0x7f6044038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6040009990 tx=0x7f6040006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.738+0000 7f604effd700 1 -- 172.21.15.2:0/2314554643 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6050019910 con 0x7f6058102770 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.930+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f605804bed0 con 0x7f6058102770 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.930+0000 7f604effd700 1 -- 172.21.15.2:0/2314554643 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f6050020020 con 0x7f6058102770 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.932+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f6044036790 msgr2=0x7f6044038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.932+0000 7f605cc09700 1 --2- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f6044036790 0x7f6044038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6040009990 tx=0x7f6040006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.932+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 msgr2=0x7f6058111530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.932+0000 7f605cc09700 1 --2- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058111530 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f60500127d0 tx=0x7f6050009840 comp rx=0 tx=0).stop 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.932+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 shutdown_connections 2024-02-16T22:12:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.932+0000 7f605cc09700 1 --2- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f6044036790 0x7f6044038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:19.502 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.932+0000 7f605cc09700 1 --2- 172.21.15.2:0/2314554643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6058102770 0x7f6058111530 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:19.502 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.933+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 >> 172.21.15.2:0/2314554643 conn(0x7f60580fded0 msgr2=0x7f60580ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:19.502 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.933+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 shutdown_connections 2024-02-16T22:12:19.502 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:18.933+0000 7f605cc09700 1 -- 172.21.15.2:0/2314554643 wait complete. 2024-02-16T22:12:19.502 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling cephadm module... 2024-02-16T22:12:21.135 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:20 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2064284665' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-16T22:12:21.419 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.904+0000 7f2d2f816700 1 Processor -- start 2024-02-16T22:12:21.419 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.905+0000 7f2d2f816700 1 -- start start 2024-02-16T22:12:21.419 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.905+0000 7f2d2f816700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28102b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:21.419 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.905+0000 7f2d2f816700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d281030d0 con 0x7f2d28102780 2024-02-16T22:12:21.419 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.905+0000 7f2d2d5b2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:21.419 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.905+0000 7f2d2d5b2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60980/0 (socket says 172.21.15.2:60980) 2024-02-16T22:12:21.419 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.905+0000 7f2d2d5b2700 1 -- 172.21.15.2:0/178026570 learned_addr learned my addr 172.21.15.2:0/178026570 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:21.419 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.905+0000 7f2d2d5b2700 1 -- 172.21.15.2:0/178026570 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28103210 con 0x7f2d28102780 2024-02-16T22:12:21.419 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.905+0000 7f2d2d5b2700 1 --2- 172.21.15.2:0/178026570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28102b90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d240098e0 tx=0x7f2d24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1310c83d19516dd7 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.905+0000 7f2d1ffff700 1 -- 172.21.15.2:0/178026570 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d2400b860 con 0x7f2d28102780 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.906+0000 7f2d1ffff700 1 -- 172.21.15.2:0/178026570 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2d2400be50 con 0x7f2d28102780 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.906+0000 7f2d1ffff700 1 -- 172.21.15.2:0/178026570 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d24010380 con 0x7f2d28102780 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.906+0000 7f2d2f816700 1 -- 172.21.15.2:0/178026570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 msgr2=0x7f2d28102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.906+0000 7f2d2f816700 1 --2- 172.21.15.2:0/178026570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28102b90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d240098e0 tx=0x7f2d24009250 comp rx=0 tx=0).stop 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.906+0000 7f2d2f816700 1 -- 172.21.15.2:0/178026570 shutdown_connections 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.906+0000 7f2d2f816700 1 --2- 172.21.15.2:0/178026570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28102b90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.906+0000 7f2d2f816700 1 -- 172.21.15.2:0/178026570 >> 172.21.15.2:0/178026570 conn(0x7f2d280fdf00 msgr2=0x7f2d28100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.906+0000 7f2d2f816700 1 -- 172.21.15.2:0/178026570 shutdown_connections 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.906+0000 7f2d2f816700 1 -- 172.21.15.2:0/178026570 wait complete. 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.907+0000 7f2d2f816700 1 Processor -- start 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.907+0000 7f2d2f816700 1 -- start start 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.907+0000 7f2d2f816700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:21.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.907+0000 7f2d2f816700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d28077190 con 0x7f2d28102780 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.907+0000 7f2d2d5b2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.907+0000 7f2d2d5b2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60982/0 (socket says 172.21.15.2:60982) 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.907+0000 7f2d2d5b2700 1 -- 172.21.15.2:0/2064284665 learned_addr learned my addr 172.21.15.2:0/2064284665 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.907+0000 7f2d2d5b2700 1 -- 172.21.15.2:0/2064284665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d240094f0 con 0x7f2d28102780 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.907+0000 7f2d2d5b2700 1 --2- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28078d10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2d24000c00 tx=0x7f2d24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.908+0000 7f2d1dffb700 1 -- 172.21.15.2:0/2064284665 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d240107c0 con 0x7f2d28102780 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.908+0000 7f2d1dffb700 1 -- 172.21.15.2:0/2064284665 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2d24010db0 con 0x7f2d28102780 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.908+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d28077390 con 0x7f2d28102780 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.908+0000 7f2d1dffb700 1 -- 172.21.15.2:0/2064284665 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d2401ad00 con 0x7f2d28102780 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.908+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d28077820 con 0x7f2d28102780 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.909+0000 7f2d1dffb700 1 -- 172.21.15.2:0/2064284665 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f2d240193c0 con 0x7f2d28102780 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.909+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2804bed0 con 0x7f2d28102780 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.909+0000 7f2d1dffb700 1 --2- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f2d14036790 0x7f2d14038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:21.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.909+0000 7f2d1dffb700 1 -- 172.21.15.2:0/2064284665 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2d2404a720 con 0x7f2d28102780 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.909+0000 7f2d2cdb1700 1 --2- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f2d14036790 0x7f2d14038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.909+0000 7f2d2cdb1700 1 --2- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f2d14036790 0x7f2d14038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009990 tx=0x7f2d18006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:19.913+0000 7f2d1dffb700 1 -- 172.21.15.2:0/2064284665 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2d24019670 con 0x7f2d28102780 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.158+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2d28077ff0 con 0x7f2d28102780 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.809+0000 7f2d1dffb700 1 -- 172.21.15.2:0/2064284665 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2d24010b00 con 0x7f2d28102780 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.811+0000 7f2d1dffb700 1 -- 172.21.15.2:0/2064284665 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f2d24023d00 con 0x7f2d28102780 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.813+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f2d14036790 msgr2=0x7f2d14038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.813+0000 7f2d2f816700 1 --2- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f2d14036790 0x7f2d14038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009990 tx=0x7f2d18006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.813+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 msgr2=0x7f2d28078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.813+0000 7f2d2f816700 1 --2- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28078d10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2d24000c00 tx=0x7f2d24009250 comp rx=0 tx=0).stop 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.813+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 shutdown_connections 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.813+0000 7f2d2f816700 1 --2- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f2d14036790 0x7f2d14038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.813+0000 7f2d2f816700 1 --2- 172.21.15.2:0/2064284665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d28102780 0x7f2d28078d10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:21.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.813+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 >> 172.21.15.2:0/2064284665 conn(0x7f2d280fdf00 msgr2=0x7f2d280fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:21.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.814+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 shutdown_connections 2024-02-16T22:12:21.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:20.814+0000 7f2d2f816700 1 -- 172.21.15.2:0/2064284665 wait complete. 2024-02-16T22:12:22.048 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:21 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2064284665' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-16T22:12:22.049 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:21 smithi002 ceph-mon[29154]: mgrmap e5: smithi002.resmjv(active, since 7s) 2024-02-16T22:12:22.683 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "smithi002.resmjv", 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.841+0000 7f82137d4700 1 Processor -- start 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.842+0000 7f82137d4700 1 -- start start 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.842+0000 7f82137d4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.842+0000 7f82137d4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c1030c0 con 0x7f820c102770 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.842+0000 7f8211570700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.842+0000 7f8211570700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57292/0 (socket says 172.21.15.2:57292) 2024-02-16T22:12:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.842+0000 7f8211570700 1 -- 172.21.15.2:0/3699473940 learned_addr learned my addr 172.21.15.2:0/3699473940 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.842+0000 7f8211570700 1 -- 172.21.15.2:0/3699473940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c103200 con 0x7f820c102770 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.842+0000 7f8211570700 1 --2- 172.21.15.2:0/3699473940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f82080098e0 tx=0x7f8208009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c5f178a1cb2b83 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.843+0000 7f81fffff700 1 -- 172.21.15.2:0/3699473940 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f820800b860 con 0x7f820c102770 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.843+0000 7f81fffff700 1 -- 172.21.15.2:0/3699473940 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f820800be50 con 0x7f820c102770 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.843+0000 7f81fffff700 1 -- 172.21.15.2:0/3699473940 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8208010380 con 0x7f820c102770 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.843+0000 7f82137d4700 1 -- 172.21.15.2:0/3699473940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 msgr2=0x7f820c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.843+0000 7f82137d4700 1 --2- 172.21.15.2:0/3699473940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f82080098e0 tx=0x7f8208009250 comp rx=0 tx=0).stop 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.843+0000 7f82137d4700 1 -- 172.21.15.2:0/3699473940 shutdown_connections 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.843+0000 7f82137d4700 1 --2- 172.21.15.2:0/3699473940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c102b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.843+0000 7f82137d4700 1 -- 172.21.15.2:0/3699473940 >> 172.21.15.2:0/3699473940 conn(0x7f820c0fded0 msgr2=0x7f820c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.844+0000 7f82137d4700 1 -- 172.21.15.2:0/3699473940 shutdown_connections 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.844+0000 7f82137d4700 1 -- 172.21.15.2:0/3699473940 wait complete. 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.844+0000 7f82137d4700 1 Processor -- start 2024-02-16T22:12:22.685 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.844+0000 7f82137d4700 1 -- start start 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.844+0000 7f82137d4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.844+0000 7f82137d4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c111a40 con 0x7f820c102770 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.844+0000 7f8211570700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.844+0000 7f8211570700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57302/0 (socket says 172.21.15.2:57302) 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.844+0000 7f8211570700 1 -- 172.21.15.2:0/255578602 learned_addr learned my addr 172.21.15.2:0/255578602 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.845+0000 7f8211570700 1 -- 172.21.15.2:0/255578602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080094f0 con 0x7f820c102770 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.845+0000 7f8211570700 1 --2- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c111500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f820800b440 tx=0x7f8208009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.845+0000 7f81fdffb700 1 -- 172.21.15.2:0/255578602 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82080107c0 con 0x7f820c102770 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.845+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c111c40 con 0x7f820c102770 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.845+0000 7f81fdffb700 1 -- 172.21.15.2:0/255578602 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8208010db0 con 0x7f820c102770 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.845+0000 7f81fdffb700 1 -- 172.21.15.2:0/255578602 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f820801ada0 con 0x7f820c102770 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.845+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c1128e0 con 0x7f820c102770 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.846+0000 7f81fdffb700 1 -- 172.21.15.2:0/255578602 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f8208010920 con 0x7f820c102770 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.846+0000 7f81fdffb700 1 --2- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f81f80367e0 0x7f81f8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:22.686 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.846+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c05f5c0 con 0x7f820c102770 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.846+0000 7f81fdffb700 1 -- 172.21.15.2:0/255578602 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f820804ac80 con 0x7f820c102770 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.846+0000 7f8210d6f700 1 -- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f81f80367e0 msgr2=0x7f81f8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/395571706 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.846+0000 7f8210d6f700 1 --2- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f81f80367e0 0x7f81f8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:21.855+0000 7f81fdffb700 1 -- 172.21.15.2:0/255578602 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f820802e440 con 0x7f820c102770 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.047+0000 7f8210d6f700 1 -- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f81f80367e0 msgr2=0x7f81f8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/395571706 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.047+0000 7f8210d6f700 1 --2- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f81f80367e0 0x7f81f8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.114+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f820c112c80 con 0x7f820c102770 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.114+0000 7f81fdffb700 1 -- 172.21.15.2:0/255578602 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f8208018350 con 0x7f820c102770 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.117+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f81f80367e0 msgr2=0x7f81f8038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.117+0000 7f82137d4700 1 --2- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f81f80367e0 0x7f81f8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.117+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 msgr2=0x7f820c111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.117+0000 7f82137d4700 1 --2- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c111500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f820800b440 tx=0x7f8208009250 comp rx=0 tx=0).stop 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.117+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 shutdown_connections 2024-02-16T22:12:22.687 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.117+0000 7f82137d4700 1 --2- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7f81f80367e0 0x7f81f8038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:22.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.117+0000 7f82137d4700 1 --2- 172.21.15.2:0/255578602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f820c102770 0x7f820c111500 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:22.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.117+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 >> 172.21.15.2:0/255578602 conn(0x7f820c0fded0 msgr2=0x7f820c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:22.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.118+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 shutdown_connections 2024-02-16T22:12:22.688 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:22.118+0000 7f82137d4700 1 -- 172.21.15.2:0/255578602 wait complete. 2024-02-16T22:12:22.688 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2024-02-16T22:12:22.688 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 5... 2024-02-16T22:12:22.928 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:22 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/255578602' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-16T22:12:27.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: Active manager daemon smithi002.resmjv restarted 2024-02-16T22:12:27.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: Activating manager daemon smithi002.resmjv 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: osdmap e2: 0 total, 0 up, 0 in 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: mgrmap e6: smithi002.resmjv(active, starting, since 0.00322473s) 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr metadata", "who": "smithi002.resmjv", "id": "smithi002.resmjv"}]: dispatch 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: Manager daemon smithi002.resmjv is now available 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:12:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:27 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:12:28.864 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:28 smithi002 ceph-mon[29154]: Found migration_current of "None". Setting to last migration. 2024-02-16T22:12:28.864 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:28 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/mirror_snapshot_schedule"}]: dispatch 2024-02-16T22:12:28.865 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:28 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/trash_purge_schedule"}]: dispatch 2024-02-16T22:12:28.865 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:28 smithi002 ceph-mon[29154]: mgrmap e7: smithi002.resmjv(active, since 1.00546s) 2024-02-16T22:12:29.166 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.124+0000 7feca81b8700 1 Processor -- start 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.125+0000 7feca81b8700 1 -- start start 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.125+0000 7feca81b8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca01077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.125+0000 7feca81b8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca0103190 con 0x7feca0076c00 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.125+0000 7feca5f54700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca01077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.125+0000 7feca5f54700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca01077a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57312/0 (socket says 172.21.15.2:57312) 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.125+0000 7feca5f54700 1 -- 172.21.15.2:0/2833742909 learned_addr learned my addr 172.21.15.2:0/2833742909 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.125+0000 7feca5f54700 1 -- 172.21.15.2:0/2833742909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca0107ce0 con 0x7feca0076c00 2024-02-16T22:12:29.167 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.126+0000 7feca5f54700 1 --2- 172.21.15.2:0/2833742909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca01077a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fec9c0098e0 tx=0x7fec9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca53e2656931042 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.126+0000 7feca4f52700 1 -- 172.21.15.2:0/2833742909 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec9c00b860 con 0x7feca0076c00 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.126+0000 7feca4f52700 1 -- 172.21.15.2:0/2833742909 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fec9c00be50 con 0x7feca0076c00 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.126+0000 7feca4f52700 1 -- 172.21.15.2:0/2833742909 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec9c010380 con 0x7feca0076c00 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.126+0000 7feca81b8700 1 -- 172.21.15.2:0/2833742909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 msgr2=0x7feca01077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.126+0000 7feca81b8700 1 --2- 172.21.15.2:0/2833742909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca01077a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fec9c0098e0 tx=0x7fec9c009250 comp rx=0 tx=0).stop 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.126+0000 7feca81b8700 1 -- 172.21.15.2:0/2833742909 shutdown_connections 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.126+0000 7feca81b8700 1 --2- 172.21.15.2:0/2833742909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca01077a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.126+0000 7feca81b8700 1 -- 172.21.15.2:0/2833742909 >> 172.21.15.2:0/2833742909 conn(0x7feca00fcdd0 msgr2=0x7feca00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.127+0000 7feca81b8700 1 -- 172.21.15.2:0/2833742909 shutdown_connections 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.127+0000 7feca81b8700 1 -- 172.21.15.2:0/2833742909 wait complete. 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.127+0000 7feca81b8700 1 Processor -- start 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.127+0000 7feca81b8700 1 -- start start 2024-02-16T22:12:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.127+0000 7feca81b8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca0113720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.127+0000 7feca81b8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca0113c60 con 0x7feca0076c00 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.127+0000 7feca5f54700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca0113720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.127+0000 7feca5f54700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca0113720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57316/0 (socket says 172.21.15.2:57316) 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.127+0000 7feca5f54700 1 -- 172.21.15.2:0/2613020534 learned_addr learned my addr 172.21.15.2:0/2613020534 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.128+0000 7feca5f54700 1 -- 172.21.15.2:0/2613020534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c0094f0 con 0x7feca0076c00 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.128+0000 7feca5f54700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca0113720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fec9c006d40 tx=0x7fec9c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.128+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec9c010760 con 0x7feca0076c00 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.128+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fec9c010d50 con 0x7feca0076c00 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.128+0000 7feca81b8700 1 -- 172.21.15.2:0/2613020534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca0113e60 con 0x7feca0076c00 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.128+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec9c024d20 con 0x7feca0076c00 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.128+0000 7feca81b8700 1 -- 172.21.15.2:0/2613020534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca0114b00 con 0x7feca0076c00 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.129+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fec9c0108c0 con 0x7feca0076c00 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.129+0000 7fec927fc700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 0x7fec8c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:29.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.129+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 --> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec8c0393a0 con 0x7fec8c0367e0 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.129+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fec9c04a760 con 0x7feca0076c00 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.129+0000 7feca5753700 1 -- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 msgr2=0x7fec8c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/395571706 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.129+0000 7feca5753700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 0x7fec8c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.330+0000 7feca5753700 1 -- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 msgr2=0x7fec8c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/395571706 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.330+0000 7feca5753700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 0x7fec8c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.731+0000 7feca5753700 1 -- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 msgr2=0x7fec8c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/395571706 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:23.731+0000 7feca5753700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 0x7fec8c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:24.532+0000 7feca5753700 1 -- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 msgr2=0x7fec8c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/395571706 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:24.532+0000 7feca5753700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 0x7fec8c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:26.133+0000 7feca5753700 1 -- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 msgr2=0x7fec8c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/395571706 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:26.133+0000 7feca5753700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 0x7fec8c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:27.547+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7fec9c02ece0 con 0x7feca0076c00 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:27.547+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 msgr2=0x7fec8c038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-16T22:12:29.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:27.547+0000 7fec927fc700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 0x7fec8c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.548+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fec9c02e440 con 0x7feca0076c00 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.548+0000 7fec927fc700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fec8c039fc0 0x7fec8c03c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.548+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec8c0393a0 con 0x7fec8c039fc0 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.549+0000 7feca5753700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fec8c039fc0 0x7fec8c03c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.549+0000 7feca5753700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fec8c039fc0 0x7fec8c03c3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fec940039d0 tx=0x7fec94009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.550+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fec8c0393a0 con 0x7fec8c039fc0 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.556+0000 7feca81b8700 1 -- 172.21.15.2:0/2613020534 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7feca01142f0 con 0x7fec8c039fc0 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.556+0000 7fec927fc700 1 -- 172.21.15.2:0/2613020534 <== mgr.14120 v2:172.21.15.2:6800/673699505 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7feca01142f0 con 0x7fec8c039fc0 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 -- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fec8c039fc0 msgr2=0x7fec8c03c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fec8c039fc0 0x7fec8c03c3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fec940039d0 tx=0x7fec94009350 comp rx=0 tx=0).stop 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 -- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 msgr2=0x7feca0113720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca0113720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fec9c006d40 tx=0x7fec9c009250 comp rx=0 tx=0).stop 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 -- 172.21.15.2:0/2613020534 shutdown_connections 2024-02-16T22:12:29.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fec8c039fc0 0x7fec8c03c3a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:29.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca0076c00 0x7feca0113720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:29.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 --2- 172.21.15.2:0/2613020534 >> [v2:172.21.15.2:6800/395571706,v1:172.21.15.2:6801/395571706] conn(0x7fec8c0367e0 0x7fec8c038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:29.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 -- 172.21.15.2:0/2613020534 >> 172.21.15.2:0/2613020534 conn(0x7feca00fcdd0 msgr2=0x7feca00fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:29.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 -- 172.21.15.2:0/2613020534 shutdown_connections 2024-02-16T22:12:29.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:28.557+0000 7feca81b8700 1 -- 172.21.15.2:0/2613020534 wait complete. 2024-02-16T22:12:29.172 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 5 is available 2024-02-16T22:12:29.172 INFO:teuthology.orchestra.run.smithi002.stdout:Setting orchestrator backend to cephadm... 2024-02-16T22:12:30.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:29 smithi002 ceph-mon[29154]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-16T22:12:30.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:29 smithi002 ceph-mon[29154]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-16T22:12:30.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:29 smithi002 ceph-mon[29154]: [16/Feb/2024:22:12:28] ENGINE Bus STARTING 2024-02-16T22:12:30.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:29 smithi002 ceph-mon[29154]: [16/Feb/2024:22:12:28] ENGINE Serving on https://172.21.15.2:7150 2024-02-16T22:12:30.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:29 smithi002 ceph-mon[29154]: [16/Feb/2024:22:12:28] ENGINE Bus STARTED 2024-02-16T22:12:30.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:29 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:30.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:29 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:12:30.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:29 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:30.346 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.552+0000 7f95d1836700 1 Processor -- start 2024-02-16T22:12:30.346 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.552+0000 7f95d1836700 1 -- start start 2024-02-16T22:12:30.346 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.552+0000 7f95d1836700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc104230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.552+0000 7f95d1836700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc104770 con 0x7f95cc103e20 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.553+0000 7f95caffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc104230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.553+0000 7f95caffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc104230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57388/0 (socket says 172.21.15.2:57388) 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.553+0000 7f95caffd700 1 -- 172.21.15.2:0/3723042737 learned_addr learned my addr 172.21.15.2:0/3723042737 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.553+0000 7f95caffd700 1 -- 172.21.15.2:0/3723042737 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc1048b0 con 0x7f95cc103e20 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.553+0000 7f95caffd700 1 --2- 172.21.15.2:0/3723042737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc104230 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0098e0 tx=0x7f95bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ea0c907c175163 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.553+0000 7f95c9ffb700 1 -- 172.21.15.2:0/3723042737 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95bc00b860 con 0x7f95cc103e20 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.553+0000 7f95c9ffb700 1 -- 172.21.15.2:0/3723042737 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f95bc00be50 con 0x7f95cc103e20 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.553+0000 7f95c9ffb700 1 -- 172.21.15.2:0/3723042737 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95bc010380 con 0x7f95cc103e20 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.554+0000 7f95d1836700 1 -- 172.21.15.2:0/3723042737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 msgr2=0x7f95cc104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:30.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.554+0000 7f95d1836700 1 --2- 172.21.15.2:0/3723042737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc104230 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0098e0 tx=0x7f95bc009250 comp rx=0 tx=0).stop 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.554+0000 7f95d1836700 1 -- 172.21.15.2:0/3723042737 shutdown_connections 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.554+0000 7f95d1836700 1 --2- 172.21.15.2:0/3723042737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc104230 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.554+0000 7f95d1836700 1 -- 172.21.15.2:0/3723042737 >> 172.21.15.2:0/3723042737 conn(0x7f95cc0fc580 msgr2=0x7f95cc0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.554+0000 7f95d1836700 1 -- 172.21.15.2:0/3723042737 shutdown_connections 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.554+0000 7f95d1836700 1 -- 172.21.15.2:0/3723042737 wait complete. 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.555+0000 7f95d1836700 1 Processor -- start 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.555+0000 7f95d1836700 1 -- start start 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.555+0000 7f95d1836700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc111810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.555+0000 7f95d1836700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc111d50 con 0x7f95cc103e20 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.555+0000 7f95caffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc111810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.555+0000 7f95caffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc111810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57398/0 (socket says 172.21.15.2:57398) 2024-02-16T22:12:30.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.555+0000 7f95caffd700 1 -- 172.21.15.2:0/3848120571 learned_addr learned my addr 172.21.15.2:0/3848120571 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.555+0000 7f95caffd700 1 -- 172.21.15.2:0/3848120571 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc0094f0 con 0x7f95cc103e20 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.555+0000 7f95caffd700 1 --2- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc111810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00b440 tx=0x7f95bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.556+0000 7f95b37fe700 1 -- 172.21.15.2:0/3848120571 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95bc0107c0 con 0x7f95cc103e20 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.556+0000 7f95b37fe700 1 -- 172.21.15.2:0/3848120571 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f95bc010db0 con 0x7f95cc103e20 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.556+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95cc111f50 con 0x7f95cc103e20 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.556+0000 7f95b37fe700 1 -- 172.21.15.2:0/3848120571 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95bc01ad80 con 0x7f95cc103e20 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.556+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95cc112bf0 con 0x7f95cc103e20 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.556+0000 7f95b37fe700 1 -- 172.21.15.2:0/3848120571 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f95bc02e440 con 0x7f95cc103e20 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.557+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95cc05f5c0 con 0x7f95cc103e20 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.557+0000 7f95b37fe700 1 --2- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f95b40366b0 0x7f95b4038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.557+0000 7f95b37fe700 1 -- 172.21.15.2:0/3848120571 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f95bc04aaf0 con 0x7f95cc103e20 2024-02-16T22:12:30.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.557+0000 7f95ca7fc700 1 --2- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f95b40366b0 0x7f95b4038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.558+0000 7f95ca7fc700 1 --2- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f95b40366b0 0x7f95b4038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009990 tx=0x7f95c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.562+0000 7f95b37fe700 1 -- 172.21.15.2:0/3848120571 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f95bc02ecb0 con 0x7f95cc103e20 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.764+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f95cc1013e0 con 0x7f95b40366b0 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.770+0000 7f95b37fe700 1 -- 172.21.15.2:0/3848120571 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f95cc1013e0 con 0x7f95b40366b0 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f95b40366b0 msgr2=0x7f95b4038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 --2- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f95b40366b0 0x7f95b4038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009990 tx=0x7f95c0006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 msgr2=0x7f95cc111810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 --2- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc111810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00b440 tx=0x7f95bc009250 comp rx=0 tx=0).stop 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 shutdown_connections 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 --2- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f95b40366b0 0x7f95b4038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 --2- 172.21.15.2:0/3848120571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc103e20 0x7f95cc111810 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:30.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 >> 172.21.15.2:0/3848120571 conn(0x7f95cc0fc580 msgr2=0x7f95cc0fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:30.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 shutdown_connections 2024-02-16T22:12:30.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:29.773+0000 7f95d1836700 1 -- 172.21.15.2:0/3848120571 wait complete. 2024-02-16T22:12:30.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:30 smithi002 ceph-mon[29154]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:30.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:30 smithi002 ceph-mon[29154]: mgrmap e8: smithi002.resmjv(active, since 2s) 2024-02-16T22:12:30.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:30 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.769+0000 7f66502a8700 1 Processor -- start 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.769+0000 7f66502a8700 1 -- start start 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.769+0000 7f66502a8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f66480771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.769+0000 7f66502a8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66480790d0 con 0x7f6648078cc0 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.770+0000 7f664e044700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f66480771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.770+0000 7f664e044700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f66480771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52586/0 (socket says 172.21.15.2:52586) 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.770+0000 7f664e044700 1 -- 172.21.15.2:0/1107249843 learned_addr learned my addr 172.21.15.2:0/1107249843 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.770+0000 7f664e044700 1 -- 172.21.15.2:0/1107249843 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6648077710 con 0x7f6648078cc0 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.770+0000 7f664e044700 1 --2- 172.21.15.2:0/1107249843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f66480771d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6640009e50 tx=0x7f6640009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a484ce9f8445307 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.770+0000 7f664d042700 1 -- 172.21.15.2:0/1107249843 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66400092e0 con 0x7f6648078cc0 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.770+0000 7f664d042700 1 -- 172.21.15.2:0/1107249843 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6640012430 con 0x7f6648078cc0 2024-02-16T22:12:31.518 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.770+0000 7f664d042700 1 -- 172.21.15.2:0/1107249843 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66400075c0 con 0x7f6648078cc0 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.771+0000 7f66502a8700 1 -- 172.21.15.2:0/1107249843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 msgr2=0x7f66480771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.771+0000 7f66502a8700 1 --2- 172.21.15.2:0/1107249843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f66480771d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6640009e50 tx=0x7f6640009840 comp rx=0 tx=0).stop 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.771+0000 7f66502a8700 1 -- 172.21.15.2:0/1107249843 shutdown_connections 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.771+0000 7f66502a8700 1 --2- 172.21.15.2:0/1107249843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f66480771d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.771+0000 7f66502a8700 1 -- 172.21.15.2:0/1107249843 >> 172.21.15.2:0/1107249843 conn(0x7f66480fded0 msgr2=0x7f6648100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.771+0000 7f66502a8700 1 -- 172.21.15.2:0/1107249843 shutdown_connections 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.771+0000 7f66502a8700 1 -- 172.21.15.2:0/1107249843 wait complete. 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f66502a8700 1 Processor -- start 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f66502a8700 1 -- start start 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f66502a8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f6648113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f66502a8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6648113d70 con 0x7f6648078cc0 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f664e044700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f6648113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f664e044700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f6648113830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52600/0 (socket says 172.21.15.2:52600) 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f664e044700 1 -- 172.21.15.2:0/2369960794 learned_addr learned my addr 172.21.15.2:0/2369960794 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:31.519 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f664e044700 1 -- 172.21.15.2:0/2369960794 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640009ae0 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f664e044700 1 --2- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f6648113830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6640009fd0 tx=0x7f6640009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f6646ffd700 1 -- 172.21.15.2:0/2369960794 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66400092e0 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f6646ffd700 1 -- 172.21.15.2:0/2369960794 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6640007cc0 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.772+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6648113f70 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.773+0000 7f6646ffd700 1 -- 172.21.15.2:0/2369960794 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f664001c850 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.773+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6648114c10 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.773+0000 7f6646ffd700 1 -- 172.21.15.2:0/2369960794 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6640019070 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.773+0000 7f6646ffd700 1 --2- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f66300367e0 0x7f6630038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.774+0000 7f6646ffd700 1 -- 172.21.15.2:0/2369960794 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6640049e40 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.774+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664805f5c0 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.774+0000 7f664d843700 1 --2- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f66300367e0 0x7f6630038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.774+0000 7f664d843700 1 --2- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f66300367e0 0x7f6630038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f663c009990 tx=0x7f663c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.779+0000 7f6646ffd700 1 -- 172.21.15.2:0/2369960794 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6640017080 con 0x7f6648078cc0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.976+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6648115730 con 0x7f66300367e0 2024-02-16T22:12:31.520 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.976+0000 7f6646ffd700 1 -- 172.21.15.2:0/2369960794 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6648115730 con 0x7f66300367e0 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f66300367e0 msgr2=0x7f6630038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 --2- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f66300367e0 0x7f6630038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f663c009990 tx=0x7f663c006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 msgr2=0x7f6648113830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 --2- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f6648113830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6640009fd0 tx=0x7f6640009840 comp rx=0 tx=0).stop 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 shutdown_connections 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 --2- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f66300367e0 0x7f6630038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 --2- 172.21.15.2:0/2369960794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6648078cc0 0x7f6648113830 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 >> 172.21.15.2:0/2369960794 conn(0x7f66480fded0 msgr2=0x7f66480ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 shutdown_connections 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:30.978+0000 7f66502a8700 1 -- 172.21.15.2:0/2369960794 wait complete. 2024-02-16T22:12:31.521 INFO:teuthology.orchestra.run.smithi002.stdout:Generating ssh key... 2024-02-16T22:12:32.181 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:31 smithi002 ceph-mon[29154]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:33.165 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:32 smithi002 ceph-mon[29154]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:33.165 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:32 smithi002 ceph-mon[29154]: Generating ssh key... 2024-02-16T22:12:33.770 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.977+0000 7fb97eff3700 1 Processor -- start 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.977+0000 7fb97eff3700 1 -- start start 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.977+0000 7fb97eff3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.977+0000 7fb97eff3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9781030c0 con 0x7fb978102770 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.978+0000 7fb97cd8f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.978+0000 7fb97cd8f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52610/0 (socket says 172.21.15.2:52610) 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.978+0000 7fb97cd8f700 1 -- 172.21.15.2:0/92766210 learned_addr learned my addr 172.21.15.2:0/92766210 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.978+0000 7fb97cd8f700 1 -- 172.21.15.2:0/92766210 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb978103200 con 0x7fb978102770 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.978+0000 7fb97cd8f700 1 --2- 172.21.15.2:0/92766210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978102b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb968009e50 tx=0x7fb968009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ccfa90c578a0b62 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.978+0000 7fb9777fe700 1 -- 172.21.15.2:0/92766210 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9680092e0 con 0x7fb978102770 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.978+0000 7fb9777fe700 1 -- 172.21.15.2:0/92766210 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb968012430 con 0x7fb978102770 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.978+0000 7fb9777fe700 1 -- 172.21.15.2:0/92766210 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9680075c0 con 0x7fb978102770 2024-02-16T22:12:33.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.979+0000 7fb97eff3700 1 -- 172.21.15.2:0/92766210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 msgr2=0x7fb978102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.979+0000 7fb97eff3700 1 --2- 172.21.15.2:0/92766210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978102b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb968009e50 tx=0x7fb968009840 comp rx=0 tx=0).stop 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.979+0000 7fb97eff3700 1 -- 172.21.15.2:0/92766210 shutdown_connections 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.979+0000 7fb97eff3700 1 --2- 172.21.15.2:0/92766210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978102b80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.979+0000 7fb97eff3700 1 -- 172.21.15.2:0/92766210 >> 172.21.15.2:0/92766210 conn(0x7fb9780fded0 msgr2=0x7fb978100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.979+0000 7fb97eff3700 1 -- 172.21.15.2:0/92766210 shutdown_connections 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.979+0000 7fb97eff3700 1 -- 172.21.15.2:0/92766210 wait complete. 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.980+0000 7fb97eff3700 1 Processor -- start 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.980+0000 7fb97eff3700 1 -- start start 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.980+0000 7fb97eff3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978111530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.980+0000 7fb97eff3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb978111a70 con 0x7fb978102770 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.980+0000 7fb97cd8f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978111530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.980+0000 7fb97cd8f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978111530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52614/0 (socket says 172.21.15.2:52614) 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.980+0000 7fb97cd8f700 1 -- 172.21.15.2:0/419648834 learned_addr learned my addr 172.21.15.2:0/419648834 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:33.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.980+0000 7fb97cd8f700 1 -- 172.21.15.2:0/419648834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb968009ae0 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.981+0000 7fb97cd8f700 1 --2- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978111530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb968000f80 tx=0x7fb968009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.981+0000 7fb9757fa700 1 -- 172.21.15.2:0/419648834 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9680092e0 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.981+0000 7fb9757fa700 1 -- 172.21.15.2:0/419648834 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb968007cc0 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.981+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb978111c70 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.981+0000 7fb9757fa700 1 -- 172.21.15.2:0/419648834 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb96801c990 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.981+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb978112910 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.982+0000 7fb9757fa700 1 -- 172.21.15.2:0/419648834 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb968019070 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.982+0000 7fb9757fa700 1 --2- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb9640367e0 0x7fb964038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.982+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97805f5c0 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.982+0000 7fb9757fa700 1 -- 172.21.15.2:0/419648834 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb968049e00 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.982+0000 7fb977fff700 1 --2- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb9640367e0 0x7fb964038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.982+0000 7fb977fff700 1 --2- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb9640367e0 0x7fb964038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb960009990 tx=0x7fb960006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:31.987+0000 7fb9757fa700 1 -- 172.21.15.2:0/419648834 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb968019cb0 con 0x7fb978102770 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:32.179+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9781001a0 con 0x7fb9640367e0 2024-02-16T22:12:33.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.168+0000 7fb9757fa700 1 -- 172.21.15.2:0/419648834 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb9781001a0 con 0x7fb9640367e0 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb9640367e0 msgr2=0x7fb964038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 --2- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb9640367e0 0x7fb964038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb960009990 tx=0x7fb960006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 msgr2=0x7fb978111530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 --2- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978111530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb968000f80 tx=0x7fb968009840 comp rx=0 tx=0).stop 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 shutdown_connections 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 --2- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb9640367e0 0x7fb964038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 --2- 172.21.15.2:0/419648834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb978102770 0x7fb978111530 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 >> 172.21.15.2:0/419648834 conn(0x7fb9780fded0 msgr2=0x7fb9780ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 shutdown_connections 2024-02-16T22:12:33.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:33.170+0000 7fb97eff3700 1 -- 172.21.15.2:0/419648834 wait complete. 2024-02-16T22:12:34.398 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:34 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:34.398 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:34 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:35.023 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCp786tCKxNJcTonI75KKDqQK7CiWwyKhVH3wEdyNzEFsnqEFqaVA763z4Q45Pw/O46QLTnfgLOyfqKVdf71mAvUaOjLRfbjIXfw+WRaVczdCC6EIoKIfYSClNZ6tYVhSlLNiFNpc6VjcRR2BDA92Q8/Hr6YABMtWeQ7GMcmdOr2pEytZM/0vETbQsziwe4JUu7Xyqxu4WzpG2EmJewk1UFAwknxKwnpe90rqaG5kt85WJ7kkeJsGC3NgzEmVM5GECEh238lOqv57nAv909nrFwQECMIVNg0QaOE+tXkLNI3xrtSryuzrgo0UdW4Iapj7qpVxhoZw9lISeO0VlreUrpZMXNJ2OJROpWgVOqvYjvn99sZQRWqqVC1BEyuz6PgR6GfM3q4IaaiIK8IfuLvyAd831SFk0a/N9+K4bvClnZVR3AY3XhkdUIbCly9HPzcmEd0dGdZ0Ijx8jFywpahh3Sshmtv6e25CIxJuo5ZCYHDrnxz90td4PNcZXxsWjeDmE= ceph-43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:12:35.023 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.195+0000 7ff78fa74700 1 Processor -- start 2024-02-16T22:12:35.023 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.195+0000 7ff78fa74700 1 -- start start 2024-02-16T22:12:35.023 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.196+0000 7ff78fa74700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff788077210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:35.023 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.196+0000 7ff78fa74700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff788079110 con 0x7ff788078d00 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.196+0000 7ff78d810700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff788077210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.196+0000 7ff78d810700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff788077210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52626/0 (socket says 172.21.15.2:52626) 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.196+0000 7ff78d810700 1 -- 172.21.15.2:0/4202835463 learned_addr learned my addr 172.21.15.2:0/4202835463 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.196+0000 7ff78d810700 1 -- 172.21.15.2:0/4202835463 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff788077750 con 0x7ff788078d00 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.196+0000 7ff78d810700 1 --2- 172.21.15.2:0/4202835463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff788077210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff7780098e0 tx=0x7ff778009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4ffe58fc47727f server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.196+0000 7ff78c80e700 1 -- 172.21.15.2:0/4202835463 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff77800b860 con 0x7ff788078d00 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.196+0000 7ff78c80e700 1 -- 172.21.15.2:0/4202835463 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff77800be50 con 0x7ff788078d00 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.197+0000 7ff78c80e700 1 -- 172.21.15.2:0/4202835463 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff778010380 con 0x7ff788078d00 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.197+0000 7ff78fa74700 1 -- 172.21.15.2:0/4202835463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 msgr2=0x7ff788077210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.197+0000 7ff78fa74700 1 --2- 172.21.15.2:0/4202835463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff788077210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff7780098e0 tx=0x7ff778009250 comp rx=0 tx=0).stop 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.197+0000 7ff78fa74700 1 -- 172.21.15.2:0/4202835463 shutdown_connections 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.197+0000 7ff78fa74700 1 --2- 172.21.15.2:0/4202835463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff788077210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.197+0000 7ff78fa74700 1 -- 172.21.15.2:0/4202835463 >> 172.21.15.2:0/4202835463 conn(0x7ff7880fded0 msgr2=0x7ff788100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.197+0000 7ff78fa74700 1 -- 172.21.15.2:0/4202835463 shutdown_connections 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.197+0000 7ff78fa74700 1 -- 172.21.15.2:0/4202835463 wait complete. 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78fa74700 1 Processor -- start 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78fa74700 1 -- start start 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78fa74700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff7881137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78fa74700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff788113d00 con 0x7ff788078d00 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78d810700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff7881137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78d810700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff7881137c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52640/0 (socket says 172.21.15.2:52640) 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78d810700 1 -- 172.21.15.2:0/135595341 learned_addr learned my addr 172.21.15.2:0/135595341 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78d810700 1 -- 172.21.15.2:0/135595341 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7780094f0 con 0x7ff788078d00 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78d810700 1 --2- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff7881137c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff778003050 tx=0x7ff778009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff77e7fc700 1 -- 172.21.15.2:0/135595341 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff778010760 con 0x7ff788078d00 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.198+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff788113f00 con 0x7ff788078d00 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.199+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff788114ba0 con 0x7ff788078d00 2024-02-16T22:12:35.025 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.199+0000 7ff77e7fc700 1 -- 172.21.15.2:0/135595341 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff778010d50 con 0x7ff788078d00 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.199+0000 7ff77e7fc700 1 -- 172.21.15.2:0/135595341 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff77801ae20 con 0x7ff788078d00 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.199+0000 7ff77e7fc700 1 -- 172.21.15.2:0/135595341 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff77802e440 con 0x7ff788078d00 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.199+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78805f5c0 con 0x7ff788078d00 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.200+0000 7ff77e7fc700 1 --2- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7740367e0 0x7ff774038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.200+0000 7ff77e7fc700 1 -- 172.21.15.2:0/135595341 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff77804a620 con 0x7ff788078d00 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.200+0000 7ff78d00f700 1 --2- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7740367e0 0x7ff774038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.200+0000 7ff78d00f700 1 --2- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7740367e0 0x7ff774038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff784009990 tx=0x7ff784006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.204+0000 7ff77e7fc700 1 -- 172.21.15.2:0/135595341 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff778018940 con 0x7ff788078d00 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.396+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff788100210 con 0x7ff7740367e0 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.397+0000 7ff77e7fc700 1 -- 172.21.15.2:0/135595341 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7ff788100210 con 0x7ff7740367e0 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7740367e0 msgr2=0x7ff774038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 --2- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7740367e0 0x7ff774038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff784009990 tx=0x7ff784006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 msgr2=0x7ff7881137c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:35.026 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 --2- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff7881137c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff778003050 tx=0x7ff778009250 comp rx=0 tx=0).stop 2024-02-16T22:12:35.027 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 shutdown_connections 2024-02-16T22:12:35.027 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 --2- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7740367e0 0x7ff774038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:35.027 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 --2- 172.21.15.2:0/135595341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff788078d00 0x7ff7881137c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:35.027 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 >> 172.21.15.2:0/135595341 conn(0x7ff7880fded0 msgr2=0x7ff7880ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:35.027 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 shutdown_connections 2024-02-16T22:12:35.027 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:34.400+0000 7ff78fa74700 1 -- 172.21.15.2:0/135595341 wait complete. 2024-02-16T22:12:35.027 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-16T22:12:35.027 INFO:teuthology.orchestra.run.smithi002.stdout:Adding key to root@localhost authorized_keys... 2024-02-16T22:12:35.027 INFO:teuthology.orchestra.run.smithi002.stdout:Adding host smithi002... 2024-02-16T22:12:36.058 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:35 smithi002 ceph-mon[29154]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:36.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:36 smithi002 ceph-mon[29154]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:38.065 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:37 smithi002 ceph-mon[29154]: Deploying cephadm binary to smithi002 2024-02-16T22:12:38.065 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:37 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:38.065 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:37 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:12:38.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Added host 'smithi002' with addr '172.21.15.2' 2024-02-16T22:12:38.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.462+0000 7f7b06f6f700 1 Processor -- start 2024-02-16T22:12:38.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.462+0000 7f7b06f6f700 1 -- start start 2024-02-16T22:12:38.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.462+0000 7f7b06f6f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b00101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.462+0000 7f7b06f6f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b00101fc0 con 0x7f7b00101670 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.462+0000 7f7b04d0b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b00101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.462+0000 7f7b04d0b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b00101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52646/0 (socket says 172.21.15.2:52646) 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.462+0000 7f7b04d0b700 1 -- 172.21.15.2:0/4154774122 learned_addr learned my addr 172.21.15.2:0/4154774122 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.462+0000 7f7b04d0b700 1 -- 172.21.15.2:0/4154774122 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00102100 con 0x7f7b00101670 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.462+0000 7f7b04d0b700 1 --2- 172.21.15.2:0/4154774122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b00101a80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7af40098e0 tx=0x7f7af4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e5ea339d517634 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7aff7fe700 1 -- 172.21.15.2:0/4154774122 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7af400b860 con 0x7f7b00101670 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7aff7fe700 1 -- 172.21.15.2:0/4154774122 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7af400be50 con 0x7f7b00101670 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7aff7fe700 1 -- 172.21.15.2:0/4154774122 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7af4010380 con 0x7f7b00101670 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7b06f6f700 1 -- 172.21.15.2:0/4154774122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 msgr2=0x7f7b00101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7b06f6f700 1 --2- 172.21.15.2:0/4154774122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b00101a80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7af40098e0 tx=0x7f7af4009250 comp rx=0 tx=0).stop 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7b06f6f700 1 -- 172.21.15.2:0/4154774122 shutdown_connections 2024-02-16T22:12:38.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7b06f6f700 1 --2- 172.21.15.2:0/4154774122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b00101a80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7b06f6f700 1 -- 172.21.15.2:0/4154774122 >> 172.21.15.2:0/4154774122 conn(0x7f7b000fcdd0 msgr2=0x7f7b000ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7b06f6f700 1 -- 172.21.15.2:0/4154774122 shutdown_connections 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.463+0000 7f7b06f6f700 1 -- 172.21.15.2:0/4154774122 wait complete. 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.464+0000 7f7b06f6f700 1 Processor -- start 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.464+0000 7f7b06f6f700 1 -- start start 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.464+0000 7f7b06f6f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b0010f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.464+0000 7f7b06f6f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0010fb60 con 0x7f7b00101670 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.464+0000 7f7b04d0b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b0010f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.464+0000 7f7b04d0b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b0010f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52658/0 (socket says 172.21.15.2:52658) 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.464+0000 7f7b04d0b700 1 -- 172.21.15.2:0/73298825 learned_addr learned my addr 172.21.15.2:0/73298825 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.464+0000 7f7b04d0b700 1 -- 172.21.15.2:0/73298825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af40094f0 con 0x7f7b00101670 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.465+0000 7f7b04d0b700 1 --2- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b0010f620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7af400b440 tx=0x7f7af4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.465+0000 7f7afd7fa700 1 -- 172.21.15.2:0/73298825 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7af4010760 con 0x7f7b00101670 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.465+0000 7f7afd7fa700 1 -- 172.21.15.2:0/73298825 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7af4010d50 con 0x7f7b00101670 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.465+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0010fd60 con 0x7f7b00101670 2024-02-16T22:12:38.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.465+0000 7f7afd7fa700 1 -- 172.21.15.2:0/73298825 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7af4024d80 con 0x7f7b00101670 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.465+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b00110a00 con 0x7f7b00101670 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.466+0000 7f7afd7fa700 1 -- 172.21.15.2:0/73298825 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7af401a450 con 0x7f7b00101670 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.466+0000 7f7afd7fa700 1 --2- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7af0038a30 0x7f7af003aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.466+0000 7f7afd7fa700 1 -- 172.21.15.2:0/73298825 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7af404a6d0 con 0x7f7b00101670 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.466+0000 7f7afffff700 1 --2- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7af0038a30 0x7f7af003aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.466+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0005f5c0 con 0x7f7b00101670 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.466+0000 7f7afffff700 1 --2- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7af0038a30 0x7f7af003aee0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009990 tx=0x7f7aec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.471+0000 7f7afd7fa700 1 -- 172.21.15.2:0/73298825 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7af402d3f0 con 0x7f7b00101670 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:35.671+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b001035f0 con 0x7f7af0038a30 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.758+0000 7f7afd7fa700 1 -- 172.21.15.2:0/73298825 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f7b001035f0 con 0x7f7af0038a30 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.760+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7af0038a30 msgr2=0x7f7af003aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.760+0000 7f7b06f6f700 1 --2- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7af0038a30 0x7f7af003aee0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009990 tx=0x7f7aec006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.760+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 msgr2=0x7f7b0010f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:38.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.760+0000 7f7b06f6f700 1 --2- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b0010f620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7af400b440 tx=0x7f7af4009250 comp rx=0 tx=0).stop 2024-02-16T22:12:38.360 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.761+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 shutdown_connections 2024-02-16T22:12:38.360 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.761+0000 7f7b06f6f700 1 --2- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7af0038a30 0x7f7af003aee0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:38.360 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.761+0000 7f7b06f6f700 1 --2- 172.21.15.2:0/73298825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b00101670 0x7f7b0010f620 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:38.360 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.761+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 >> 172.21.15.2:0/73298825 conn(0x7f7b000fcdd0 msgr2=0x7f7b001063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:38.360 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.761+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 shutdown_connections 2024-02-16T22:12:38.360 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:37.761+0000 7f7b06f6f700 1 -- 172.21.15.2:0/73298825 wait complete. 2024-02-16T22:12:38.360 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying mon service with default placement... 2024-02-16T22:12:38.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:38 smithi002 ceph-mon[29154]: Added host smithi002 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.752+0000 7f34d8c0e700 1 Processor -- start 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.752+0000 7f34d8c0e700 1 -- start start 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.752+0000 7f34d8c0e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d41077c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.752+0000 7f34d8c0e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d41031b0 con 0x7f34d4076be0 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.753+0000 7f34d259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d41077c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.753+0000 7f34d259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d41077c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52664/0 (socket says 172.21.15.2:52664) 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.753+0000 7f34d259c700 1 -- 172.21.15.2:0/1010346658 learned_addr learned my addr 172.21.15.2:0/1010346658 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.753+0000 7f34d259c700 1 -- 172.21.15.2:0/1010346658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d4107d00 con 0x7f34d4076be0 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.753+0000 7f34d259c700 1 --2- 172.21.15.2:0/1010346658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d41077c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009e50 tx=0x7f34c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1681cdab6dc48ed0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.753+0000 7f34d159a700 1 -- 172.21.15.2:0/1010346658 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c40092e0 con 0x7f34d4076be0 2024-02-16T22:12:39.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.753+0000 7f34d159a700 1 -- 172.21.15.2:0/1010346658 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f34c4012430 con 0x7f34d4076be0 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.753+0000 7f34d159a700 1 -- 172.21.15.2:0/1010346658 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c40075c0 con 0x7f34d4076be0 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.754+0000 7f34d8c0e700 1 -- 172.21.15.2:0/1010346658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 msgr2=0x7f34d41077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.754+0000 7f34d8c0e700 1 --2- 172.21.15.2:0/1010346658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d41077c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009e50 tx=0x7f34c4009840 comp rx=0 tx=0).stop 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.754+0000 7f34d8c0e700 1 -- 172.21.15.2:0/1010346658 shutdown_connections 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.754+0000 7f34d8c0e700 1 --2- 172.21.15.2:0/1010346658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d41077c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.754+0000 7f34d8c0e700 1 -- 172.21.15.2:0/1010346658 >> 172.21.15.2:0/1010346658 conn(0x7f34d40fcdd0 msgr2=0x7f34d40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.754+0000 7f34d8c0e700 1 -- 172.21.15.2:0/1010346658 shutdown_connections 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.754+0000 7f34d8c0e700 1 -- 172.21.15.2:0/1010346658 wait complete. 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.754+0000 7f34d8c0e700 1 Processor -- start 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.754+0000 7f34d8c0e700 1 -- start start 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34d8c0e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d4113760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34d8c0e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4113ca0 con 0x7f34d4076be0 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34d259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d4113760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34d259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d4113760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52672/0 (socket says 172.21.15.2:52672) 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34d259c700 1 -- 172.21.15.2:0/3822411545 learned_addr learned my addr 172.21.15.2:0/3822411545 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:39.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34d259c700 1 -- 172.21.15.2:0/3822411545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c4009ae0 con 0x7f34d4076be0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34d259c700 1 --2- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d4113760 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009ab0 tx=0x7f34c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34caffd700 1 -- 172.21.15.2:0/3822411545 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c40092e0 con 0x7f34d4076be0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34caffd700 1 -- 172.21.15.2:0/3822411545 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f34c4007d10 con 0x7f34d4076be0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d4113ea0 con 0x7f34d4076be0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34caffd700 1 -- 172.21.15.2:0/3822411545 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c401c9a0 con 0x7f34d4076be0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.755+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d4114b40 con 0x7f34d4076be0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.756+0000 7f34caffd700 1 -- 172.21.15.2:0/3822411545 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f34c4019070 con 0x7f34d4076be0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.756+0000 7f34caffd700 1 --2- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f34c00367e0 0x7f34c0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.756+0000 7f34caffd700 1 -- 172.21.15.2:0/3822411545 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f34c4049dc0 con 0x7f34d4076be0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.756+0000 7f34d1d9b700 1 --2- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f34c00367e0 0x7f34c0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.756+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d405f5c0 con 0x7f34d4076be0 2024-02-16T22:12:39.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.757+0000 7f34d1d9b700 1 --2- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f34c00367e0 0x7f34c0038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009990 tx=0x7f34bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.761+0000 7f34caffd700 1 -- 172.21.15.2:0/3822411545 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f34c4017080 con 0x7f34d4076be0 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.975+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f34d40ff090 con 0x7f34c00367e0 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.979+0000 7f34caffd700 1 -- 172.21.15.2:0/3822411545 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f34d40ff090 con 0x7f34c00367e0 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f34c00367e0 msgr2=0x7f34c0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 --2- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f34c00367e0 0x7f34c0038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009990 tx=0x7f34bc006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 msgr2=0x7f34d4113760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 --2- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d4113760 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009ab0 tx=0x7f34c4009840 comp rx=0 tx=0).stop 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 shutdown_connections 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 --2- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f34c00367e0 0x7f34c0038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 --2- 172.21.15.2:0/3822411545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34d4076be0 0x7f34d4113760 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 >> 172.21.15.2:0/3822411545 conn(0x7f34d40fcdd0 msgr2=0x7f34d40fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 shutdown_connections 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:38.981+0000 7f34d8c0e700 1 -- 172.21.15.2:0/3822411545 wait complete. 2024-02-16T22:12:39.760 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying mgr service with default placement... 2024-02-16T22:12:39.999 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:39 smithi002 ceph-mon[29154]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:39.999 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:39 smithi002 ceph-mon[29154]: Saving service mon spec with placement count:5 2024-02-16T22:12:39.999 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:39 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:41.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:41 smithi002 ceph-mon[29154]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:41.728 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:41 smithi002 ceph-mon[29154]: Saving service mgr spec with placement count:2 2024-02-16T22:12:41.728 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:41 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:41.827 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-16T22:12:41.827 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.205+0000 7ff806dc2700 1 Processor -- start 2024-02-16T22:12:41.827 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.205+0000 7ff806dc2700 1 -- start start 2024-02-16T22:12:41.827 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.205+0000 7ff806dc2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff800103050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.205+0000 7ff806dc2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff800103590 con 0x7ff800102c40 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.205+0000 7ff804b5e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff800103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.205+0000 7ff804b5e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff800103050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52686/0 (socket says 172.21.15.2:52686) 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.205+0000 7ff804b5e700 1 -- 172.21.15.2:0/1104813855 learned_addr learned my addr 172.21.15.2:0/1104813855 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.205+0000 7ff804b5e700 1 -- 172.21.15.2:0/1104813855 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8001036d0 con 0x7ff800102c40 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.205+0000 7ff804b5e700 1 --2- 172.21.15.2:0/1104813855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff800103050 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400bb80 tx=0x7ff7f400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7b5e217ad72bc9 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff7ff7fe700 1 -- 172.21.15.2:0/1104813855 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f400dd00 con 0x7ff800102c40 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff7ff7fe700 1 -- 172.21.15.2:0/1104813855 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff7f400de60 con 0x7ff800102c40 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff7ff7fe700 1 -- 172.21.15.2:0/1104813855 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f40188a0 con 0x7ff800102c40 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff806dc2700 1 -- 172.21.15.2:0/1104813855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 msgr2=0x7ff800103050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff806dc2700 1 --2- 172.21.15.2:0/1104813855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff800103050 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400bb80 tx=0x7ff7f400b4f0 comp rx=0 tx=0).stop 2024-02-16T22:12:41.828 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff806dc2700 1 -- 172.21.15.2:0/1104813855 shutdown_connections 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff806dc2700 1 --2- 172.21.15.2:0/1104813855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff800103050 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff806dc2700 1 -- 172.21.15.2:0/1104813855 >> 172.21.15.2:0/1104813855 conn(0x7ff8000fc580 msgr2=0x7ff8000fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff806dc2700 1 -- 172.21.15.2:0/1104813855 shutdown_connections 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.206+0000 7ff806dc2700 1 -- 172.21.15.2:0/1104813855 wait complete. 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.207+0000 7ff806dc2700 1 Processor -- start 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.207+0000 7ff806dc2700 1 -- start start 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.207+0000 7ff806dc2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff80010f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.207+0000 7ff806dc2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80010faf0 con 0x7ff800102c40 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.207+0000 7ff804b5e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff80010f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.207+0000 7ff804b5e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff80010f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52702/0 (socket says 172.21.15.2:52702) 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.207+0000 7ff804b5e700 1 -- 172.21.15.2:0/2827717356 learned_addr learned my addr 172.21.15.2:0/2827717356 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.208+0000 7ff804b5e700 1 -- 172.21.15.2:0/2827717356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f400b790 con 0x7ff800102c40 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.208+0000 7ff804b5e700 1 --2- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff80010f5b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4010fa0 tx=0x7ff7f400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.208+0000 7ff7fd7fa700 1 -- 172.21.15.2:0/2827717356 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f4010b90 con 0x7ff800102c40 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.208+0000 7ff7fd7fa700 1 -- 172.21.15.2:0/2827717356 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff7f401d050 con 0x7ff800102c40 2024-02-16T22:12:41.829 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.208+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80010fcf0 con 0x7ff800102c40 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.208+0000 7ff7fd7fa700 1 -- 172.21.15.2:0/2827717356 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f4022590 con 0x7ff800102c40 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.208+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff800110990 con 0x7ff800102c40 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.209+0000 7ff7fd7fa700 1 -- 172.21.15.2:0/2827717356 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff7f4023470 con 0x7ff800102c40 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.209+0000 7ff7fd7fa700 1 --2- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7f00367e0 0x7ff7f0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.209+0000 7ff7fd7fa700 1 -- 172.21.15.2:0/2827717356 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff7f4014070 con 0x7ff800102c40 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.209+0000 7ff7fffff700 1 --2- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7f00367e0 0x7ff7f0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.209+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff800110180 con 0x7ff800102c40 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.210+0000 7ff7fffff700 1 --2- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7f00367e0 0x7ff7f0038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009990 tx=0x7ff7ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.214+0000 7ff7fd7fa700 1 -- 172.21.15.2:0/2827717356 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff7f4022740 con 0x7ff800102c40 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.421+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7ff800101460 con 0x7ff7f00367e0 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.425+0000 7ff7fd7fa700 1 -- 172.21.15.2:0/2827717356 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff800101460 con 0x7ff7f00367e0 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7f00367e0 msgr2=0x7ff7f0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:41.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 --2- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7f00367e0 0x7ff7f0038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009990 tx=0x7ff7ec006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:41.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 msgr2=0x7ff80010f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:41.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 --2- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff80010f5b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4010fa0 tx=0x7ff7f400b4f0 comp rx=0 tx=0).stop 2024-02-16T22:12:41.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 shutdown_connections 2024-02-16T22:12:41.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 --2- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7ff7f00367e0 0x7ff7f0038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:41.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 --2- 172.21.15.2:0/2827717356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff800102c40 0x7ff80010f5b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:41.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 >> 172.21.15.2:0/2827717356 conn(0x7ff8000fc580 msgr2=0x7ff800106470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:41.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 shutdown_connections 2024-02-16T22:12:41.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:40.427+0000 7ff806dc2700 1 -- 172.21.15.2:0/2827717356 wait complete. 2024-02-16T22:12:41.831 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying crash service with default placement... 2024-02-16T22:12:42.451 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:42 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:42.451 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:42 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.249+0000 7f7f73092700 1 Processor -- start 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.249+0000 7f7f73092700 1 -- start start 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.249+0000 7f7f73092700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.249+0000 7f7f73092700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c1030c0 con 0x7f7f6c102770 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.250+0000 7f7f70e2e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.250+0000 7f7f70e2e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52852/0 (socket says 172.21.15.2:52852) 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.250+0000 7f7f70e2e700 1 -- 172.21.15.2:0/3674839672 learned_addr learned my addr 172.21.15.2:0/3674839672 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.250+0000 7f7f70e2e700 1 -- 172.21.15.2:0/3674839672 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c103200 con 0x7f7f6c102770 2024-02-16T22:12:43.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.250+0000 7f7f70e2e700 1 --2- 172.21.15.2:0/3674839672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c102b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f600098e0 tx=0x7f7f60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3141a2525309d92c server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.250+0000 7f7f6b7fe700 1 -- 172.21.15.2:0/3674839672 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f6000b860 con 0x7f7f6c102770 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.250+0000 7f7f6b7fe700 1 -- 172.21.15.2:0/3674839672 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f6000be90 con 0x7f7f6c102770 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.250+0000 7f7f6b7fe700 1 -- 172.21.15.2:0/3674839672 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f600103c0 con 0x7f7f6c102770 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.251+0000 7f7f73092700 1 -- 172.21.15.2:0/3674839672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 msgr2=0x7f7f6c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.251+0000 7f7f73092700 1 --2- 172.21.15.2:0/3674839672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c102b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f600098e0 tx=0x7f7f60009250 comp rx=0 tx=0).stop 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.251+0000 7f7f73092700 1 -- 172.21.15.2:0/3674839672 shutdown_connections 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.251+0000 7f7f73092700 1 --2- 172.21.15.2:0/3674839672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c102b80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.251+0000 7f7f73092700 1 -- 172.21.15.2:0/3674839672 >> 172.21.15.2:0/3674839672 conn(0x7f7f6c0fded0 msgr2=0x7f7f6c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.251+0000 7f7f73092700 1 -- 172.21.15.2:0/3674839672 shutdown_connections 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.251+0000 7f7f73092700 1 -- 172.21.15.2:0/3674839672 wait complete. 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.251+0000 7f7f73092700 1 Processor -- start 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.251+0000 7f7f73092700 1 -- start start 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f73092700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c1115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:43.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f73092700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c111af0 con 0x7f7f6c102770 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f70e2e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c1115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f70e2e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c1115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52862/0 (socket says 172.21.15.2:52862) 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f70e2e700 1 -- 172.21.15.2:0/1810367998 learned_addr learned my addr 172.21.15.2:0/1810367998 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f70e2e700 1 -- 172.21.15.2:0/1810367998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f600094f0 con 0x7f7f6c102770 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f70e2e700 1 --2- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c1115b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7f600090b0 tx=0x7f7f60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f697fa700 1 -- 172.21.15.2:0/1810367998 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f60010840 con 0x7f7f6c102770 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f697fa700 1 -- 172.21.15.2:0/1810367998 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f60010e70 con 0x7f7f6c102770 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f697fa700 1 -- 172.21.15.2:0/1810367998 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f60024e40 con 0x7f7f6c102770 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c111cf0 con 0x7f7f6c102770 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.252+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c112990 con 0x7f7f6c102770 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.253+0000 7f7f697fa700 1 -- 172.21.15.2:0/1810367998 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7f6001a4e0 con 0x7f7f6c102770 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.253+0000 7f7f697fa700 1 --2- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7f5c036790 0x7f7f5c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.253+0000 7f7f697fa700 1 -- 172.21.15.2:0/1810367998 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7f6004a770 con 0x7f7f6c102770 2024-02-16T22:12:43.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.253+0000 7f7f6bfff700 1 --2- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7f5c036790 0x7f7f5c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.253+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c112180 con 0x7f7f6c102770 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.254+0000 7f7f6bfff700 1 --2- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7f5c036790 0x7f7f5c038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009990 tx=0x7f7f58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.258+0000 7f7f697fa700 1 -- 172.21.15.2:0/1810367998 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7f60018940 con 0x7f7f6c102770 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.474+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f6c1001b0 con 0x7f7f5c036790 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.479+0000 7f7f697fa700 1 -- 172.21.15.2:0/1810367998 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f7f6c1001b0 con 0x7f7f5c036790 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7f5c036790 msgr2=0x7f7f5c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 --2- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7f5c036790 0x7f7f5c038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009990 tx=0x7f7f58006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 msgr2=0x7f7f6c1115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 --2- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c1115b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7f600090b0 tx=0x7f7f60009250 comp rx=0 tx=0).stop 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 shutdown_connections 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 --2- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7f5c036790 0x7f7f5c038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 --2- 172.21.15.2:0/1810367998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f6c102770 0x7f7f6c1115b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 >> 172.21.15.2:0/1810367998 conn(0x7f7f6c0fded0 msgr2=0x7f7f6c0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 shutdown_connections 2024-02-16T22:12:43.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:42.481+0000 7f7f73092700 1 -- 172.21.15.2:0/1810367998 wait complete. 2024-02-16T22:12:43.323 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying prometheus service with default placement... 2024-02-16T22:12:43.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:43 smithi002 ceph-mon[29154]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:43.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:43 smithi002 ceph-mon[29154]: Saving service crash spec with placement * 2024-02-16T22:12:43.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:43 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:44.961 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:44 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:44.962 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:44 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:44.962 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:44 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:44.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-16T22:12:44.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.085+0000 7f7313d81700 1 Processor -- start 2024-02-16T22:12:44.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.085+0000 7f7313d81700 1 -- start start 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.085+0000 7f7313d81700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c105320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.085+0000 7f7313d81700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c105860 con 0x7f730c076c20 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.085+0000 7f7311b1d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c105320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.085+0000 7f7311b1d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c105320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52874/0 (socket says 172.21.15.2:52874) 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.085+0000 7f7311b1d700 1 -- 172.21.15.2:0/422207037 learned_addr learned my addr 172.21.15.2:0/422207037 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.085+0000 7f7311b1d700 1 -- 172.21.15.2:0/422207037 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c1059a0 con 0x7f730c076c20 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.085+0000 7f7311b1d700 1 --2- 172.21.15.2:0/422207037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c105320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0098e0 tx=0x7f72fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e5e7acce064f6d server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7310b1b700 1 -- 172.21.15.2:0/422207037 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72fc00b860 con 0x7f730c076c20 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7310b1b700 1 -- 172.21.15.2:0/422207037 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72fc00be90 con 0x7f730c076c20 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7310b1b700 1 -- 172.21.15.2:0/422207037 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72fc0103c0 con 0x7f730c076c20 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7313d81700 1 -- 172.21.15.2:0/422207037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 msgr2=0x7f730c105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7313d81700 1 --2- 172.21.15.2:0/422207037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c105320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0098e0 tx=0x7f72fc009250 comp rx=0 tx=0).stop 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7313d81700 1 -- 172.21.15.2:0/422207037 shutdown_connections 2024-02-16T22:12:44.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7313d81700 1 --2- 172.21.15.2:0/422207037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c105320 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7313d81700 1 -- 172.21.15.2:0/422207037 >> 172.21.15.2:0/422207037 conn(0x7f730c0fcd50 msgr2=0x7f730c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7313d81700 1 -- 172.21.15.2:0/422207037 shutdown_connections 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.086+0000 7f7313d81700 1 -- 172.21.15.2:0/422207037 wait complete. 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.087+0000 7f7313d81700 1 Processor -- start 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.087+0000 7f7313d81700 1 -- start start 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.087+0000 7f7313d81700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c1114f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.087+0000 7f7313d81700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c111a30 con 0x7f730c076c20 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.087+0000 7f7311b1d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c1114f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.087+0000 7f7311b1d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c1114f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52882/0 (socket says 172.21.15.2:52882) 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.087+0000 7f7311b1d700 1 -- 172.21.15.2:0/3633034209 learned_addr learned my addr 172.21.15.2:0/3633034209 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.088+0000 7f7311b1d700 1 -- 172.21.15.2:0/3633034209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc0094f0 con 0x7f730c076c20 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.088+0000 7f7311b1d700 1 --2- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c1114f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0090b0 tx=0x7f72fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.088+0000 7f73027fc700 1 -- 172.21.15.2:0/3633034209 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72fc0118f0 con 0x7f730c076c20 2024-02-16T22:12:44.971 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.088+0000 7f73027fc700 1 -- 172.21.15.2:0/3633034209 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72fc010d10 con 0x7f730c076c20 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.088+0000 7f73027fc700 1 -- 172.21.15.2:0/3633034209 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72fc024e70 con 0x7f730c076c20 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.088+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c111c30 con 0x7f730c076c20 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.088+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c1128d0 con 0x7f730c076c20 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.089+0000 7f73027fc700 1 -- 172.21.15.2:0/3633034209 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f72fc024440 con 0x7f730c076c20 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.089+0000 7f73027fc700 1 --2- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f72f8036740 0x7f72f8038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.089+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c05f5c0 con 0x7f730c076c20 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.089+0000 7f73027fc700 1 -- 172.21.15.2:0/3633034209 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f72fc04a550 con 0x7f730c076c20 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.089+0000 7f731131c700 1 --2- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f72f8036740 0x7f72f8038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.089+0000 7f731131c700 1 --2- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f72f8036740 0x7f72f8038bf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7308009990 tx=0x7f7308006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.094+0000 7f73027fc700 1 -- 172.21.15.2:0/3633034209 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f72fc022930 con 0x7f730c076c20 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.334+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f730c0ff050 con 0x7f72f8036740 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.339+0000 7f73027fc700 1 -- 172.21.15.2:0/3633034209 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f730c0ff050 con 0x7f72f8036740 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f72f8036740 msgr2=0x7f72f8038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:44.972 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 --2- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f72f8036740 0x7f72f8038bf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7308009990 tx=0x7f7308006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:44.973 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 msgr2=0x7f730c1114f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:44.973 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 --2- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c1114f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0090b0 tx=0x7f72fc009250 comp rx=0 tx=0).stop 2024-02-16T22:12:44.973 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 shutdown_connections 2024-02-16T22:12:44.973 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 --2- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f72f8036740 0x7f72f8038bf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:44.973 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 --2- 172.21.15.2:0/3633034209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f730c076c20 0x7f730c1114f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:44.973 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 >> 172.21.15.2:0/3633034209 conn(0x7f730c0fcd50 msgr2=0x7f730c0fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:44.973 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 shutdown_connections 2024-02-16T22:12:44.973 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:44.341+0000 7f7313d81700 1 -- 172.21.15.2:0/3633034209 wait complete. 2024-02-16T22:12:44.973 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying grafana service with default placement... 2024-02-16T22:12:45.907 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:45 smithi002 ceph-mon[29154]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:45.908 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:45 smithi002 ceph-mon[29154]: Saving service prometheus spec with placement count:1 2024-02-16T22:12:45.908 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:45 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:46.957 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-16T22:12:46.957 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.369+0000 7f5e3a69c700 1 Processor -- start 2024-02-16T22:12:46.957 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.369+0000 7f5e3a69c700 1 -- start start 2024-02-16T22:12:46.957 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.369+0000 7f5e3a69c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34102b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:46.957 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.369+0000 7f5e3a69c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e341030d0 con 0x7f5e34102780 2024-02-16T22:12:46.957 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.370+0000 7f5e33fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.370+0000 7f5e33fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52896/0 (socket says 172.21.15.2:52896) 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.370+0000 7f5e33fff700 1 -- 172.21.15.2:0/2088258403 learned_addr learned my addr 172.21.15.2:0/2088258403 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.370+0000 7f5e33fff700 1 -- 172.21.15.2:0/2088258403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e34103210 con 0x7f5e34102780 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.370+0000 7f5e33fff700 1 --2- 172.21.15.2:0/2088258403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34102b90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e240098e0 tx=0x7f5e24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32b682620dae5ab server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.370+0000 7f5e32ffd700 1 -- 172.21.15.2:0/2088258403 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e2400b860 con 0x7f5e34102780 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.370+0000 7f5e32ffd700 1 -- 172.21.15.2:0/2088258403 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e2400be90 con 0x7f5e34102780 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.370+0000 7f5e32ffd700 1 -- 172.21.15.2:0/2088258403 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e240103c0 con 0x7f5e34102780 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.371+0000 7f5e3a69c700 1 -- 172.21.15.2:0/2088258403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 msgr2=0x7f5e34102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.371+0000 7f5e3a69c700 1 --2- 172.21.15.2:0/2088258403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34102b90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e240098e0 tx=0x7f5e24009250 comp rx=0 tx=0).stop 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.371+0000 7f5e3a69c700 1 -- 172.21.15.2:0/2088258403 shutdown_connections 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.371+0000 7f5e3a69c700 1 --2- 172.21.15.2:0/2088258403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34102b90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.371+0000 7f5e3a69c700 1 -- 172.21.15.2:0/2088258403 >> 172.21.15.2:0/2088258403 conn(0x7f5e340fdf00 msgr2=0x7f5e34100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.371+0000 7f5e3a69c700 1 -- 172.21.15.2:0/2088258403 shutdown_connections 2024-02-16T22:12:46.958 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.371+0000 7f5e3a69c700 1 -- 172.21.15.2:0/2088258403 wait complete. 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.371+0000 7f5e3a69c700 1 Processor -- start 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.372+0000 7f5e3a69c700 1 -- start start 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.372+0000 7f5e3a69c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.372+0000 7f5e3a69c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e34111ad0 con 0x7f5e34102780 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.372+0000 7f5e33fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.372+0000 7f5e33fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52908/0 (socket says 172.21.15.2:52908) 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.372+0000 7f5e33fff700 1 -- 172.21.15.2:0/3966028704 learned_addr learned my addr 172.21.15.2:0/3966028704 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.372+0000 7f5e33fff700 1 -- 172.21.15.2:0/3966028704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e240094f0 con 0x7f5e34102780 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.372+0000 7f5e33fff700 1 --2- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34111590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5e24011680 tx=0x7f5e24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.372+0000 7f5e30ff9700 1 -- 172.21.15.2:0/3966028704 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e24010890 con 0x7f5e34102780 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.373+0000 7f5e30ff9700 1 -- 172.21.15.2:0/3966028704 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e24010ec0 con 0x7f5e34102780 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.373+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e34111cd0 con 0x7f5e34102780 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.373+0000 7f5e30ff9700 1 -- 172.21.15.2:0/3966028704 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e240243f0 con 0x7f5e34102780 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.373+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e34112970 con 0x7f5e34102780 2024-02-16T22:12:46.959 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.373+0000 7f5e30ff9700 1 -- 172.21.15.2:0/3966028704 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5e2402d3f0 con 0x7f5e34102780 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.373+0000 7f5e30ff9700 1 --2- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f5e1c0367e0 0x7f5e1c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.373+0000 7f5e30ff9700 1 -- 172.21.15.2:0/3966028704 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5e2404b2d0 con 0x7f5e34102780 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.374+0000 7f5e337fe700 1 --2- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f5e1c0367e0 0x7f5e1c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.374+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e34112160 con 0x7f5e34102780 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.374+0000 7f5e337fe700 1 --2- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f5e1c0367e0 0x7f5e1c038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009990 tx=0x7f5e28006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.378+0000 7f5e30ff9700 1 -- 172.21.15.2:0/3966028704 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5e2402b8e0 con 0x7f5e34102780 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.595+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e341001b0 con 0x7f5e1c0367e0 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.599+0000 7f5e30ff9700 1 -- 172.21.15.2:0/3966028704 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5e341001b0 con 0x7f5e1c0367e0 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f5e1c0367e0 msgr2=0x7f5e1c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 --2- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f5e1c0367e0 0x7f5e1c038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009990 tx=0x7f5e28006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 msgr2=0x7f5e34111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 --2- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34111590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5e24011680 tx=0x7f5e24009250 comp rx=0 tx=0).stop 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 shutdown_connections 2024-02-16T22:12:46.960 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 --2- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f5e1c0367e0 0x7f5e1c038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:46.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 --2- 172.21.15.2:0/3966028704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e34102780 0x7f5e34111590 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:46.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 >> 172.21.15.2:0/3966028704 conn(0x7f5e340fdf00 msgr2=0x7f5e340ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:46.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 shutdown_connections 2024-02-16T22:12:46.961 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:45.601+0000 7f5e3a69c700 1 -- 172.21.15.2:0/3966028704 wait complete. 2024-02-16T22:12:46.961 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying node-exporter service with default placement... 2024-02-16T22:12:47.080 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:46 smithi002 ceph-mon[29154]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:47.080 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:46 smithi002 ceph-mon[29154]: Saving service grafana spec with placement count:1 2024-02-16T22:12:48.180 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.397+0000 7f7d2dadc700 1 Processor -- start 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.397+0000 7f7d2dadc700 1 -- start start 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.397+0000 7f7d2dadc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d28102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.397+0000 7f7d2dadc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d281030c0 con 0x7f7d28102770 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.397+0000 7f7d277fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d28102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.397+0000 7f7d277fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d28102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52914/0 (socket says 172.21.15.2:52914) 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.397+0000 7f7d277fe700 1 -- 172.21.15.2:0/1930380459 learned_addr learned my addr 172.21.15.2:0/1930380459 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.397+0000 7f7d277fe700 1 -- 172.21.15.2:0/1930380459 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d28103200 con 0x7f7d28102770 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.397+0000 7f7d277fe700 1 --2- 172.21.15.2:0/1930380459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d28102b80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0098e0 tx=0x7f7d1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658d29a81720b2f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d267fc700 1 -- 172.21.15.2:0/1930380459 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d1c00b860 con 0x7f7d28102770 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d267fc700 1 -- 172.21.15.2:0/1930380459 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d1c00be90 con 0x7f7d28102770 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d267fc700 1 -- 172.21.15.2:0/1930380459 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d1c0103c0 con 0x7f7d28102770 2024-02-16T22:12:48.181 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1930380459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 msgr2=0x7f7d28102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d2dadc700 1 --2- 172.21.15.2:0/1930380459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d28102b80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0098e0 tx=0x7f7d1c009250 comp rx=0 tx=0).stop 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1930380459 shutdown_connections 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d2dadc700 1 --2- 172.21.15.2:0/1930380459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d28102b80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1930380459 >> 172.21.15.2:0/1930380459 conn(0x7f7d280fded0 msgr2=0x7f7d28100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1930380459 shutdown_connections 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.398+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1930380459 wait complete. 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.399+0000 7f7d2dadc700 1 Processor -- start 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.399+0000 7f7d2dadc700 1 -- start start 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.399+0000 7f7d2dadc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d281115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.399+0000 7f7d2dadc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d28111af0 con 0x7f7d28102770 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.399+0000 7f7d277fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d281115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.399+0000 7f7d277fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d281115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52924/0 (socket says 172.21.15.2:52924) 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.399+0000 7f7d277fe700 1 -- 172.21.15.2:0/1498006167 learned_addr learned my addr 172.21.15.2:0/1498006167 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.400+0000 7f7d277fe700 1 -- 172.21.15.2:0/1498006167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c0094f0 con 0x7f7d28102770 2024-02-16T22:12:48.182 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.400+0000 7f7d277fe700 1 --2- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d281115b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c011680 tx=0x7f7d1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.400+0000 7f7d0ffff700 1 -- 172.21.15.2:0/1498006167 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d1c010890 con 0x7f7d28102770 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.400+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d28111cf0 con 0x7f7d28102770 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.400+0000 7f7d0ffff700 1 -- 172.21.15.2:0/1498006167 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d1c010ec0 con 0x7f7d28102770 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.400+0000 7f7d0ffff700 1 -- 172.21.15.2:0/1498006167 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d1c0243f0 con 0x7f7d28102770 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.400+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d28112990 con 0x7f7d28102770 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.401+0000 7f7d0ffff700 1 -- 172.21.15.2:0/1498006167 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7d1c02d3f0 con 0x7f7d28102770 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.401+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2805f5c0 con 0x7f7d28102770 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.401+0000 7f7d0ffff700 1 --2- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7d100367e0 0x7f7d10038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.401+0000 7f7d0ffff700 1 -- 172.21.15.2:0/1498006167 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7d1c04a260 con 0x7f7d28102770 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.401+0000 7f7d26ffd700 1 --2- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7d100367e0 0x7f7d10038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.402+0000 7f7d26ffd700 1 --2- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7d100367e0 0x7f7d10038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009990 tx=0x7f7d18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.406+0000 7f7d0ffff700 1 -- 172.21.15.2:0/1498006167 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7d1c02b8e0 con 0x7f7d28102770 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.608+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d281001b0 con 0x7f7d100367e0 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.612+0000 7f7d0ffff700 1 -- 172.21.15.2:0/1498006167 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f7d281001b0 con 0x7f7d100367e0 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7d100367e0 msgr2=0x7f7d10038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:48.183 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 --2- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7d100367e0 0x7f7d10038c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009990 tx=0x7f7d18006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:48.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 msgr2=0x7f7d281115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:48.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 --2- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d281115b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c011680 tx=0x7f7d1c009250 comp rx=0 tx=0).stop 2024-02-16T22:12:48.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 shutdown_connections 2024-02-16T22:12:48.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 --2- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f7d100367e0 0x7f7d10038c90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:48.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 --2- 172.21.15.2:0/1498006167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d28102770 0x7f7d281115b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:48.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 >> 172.21.15.2:0/1498006167 conn(0x7f7d280fded0 msgr2=0x7f7d280ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:48.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 shutdown_connections 2024-02-16T22:12:48.184 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:47.614+0000 7f7d2dadc700 1 -- 172.21.15.2:0/1498006167 wait complete. 2024-02-16T22:12:48.184 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying alertmanager service with default placement... 2024-02-16T22:12:48.805 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:48 smithi002 ceph-mon[29154]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:12:48.805 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:48 smithi002 ceph-mon[29154]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:48.805 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:48 smithi002 ceph-mon[29154]: Saving service node-exporter spec with placement * 2024-02-16T22:12:48.805 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:48 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:49.527 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.590+0000 7f6167713700 1 Processor -- start 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.590+0000 7f6167713700 1 -- start start 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.590+0000 7f6167713700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f6160102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.590+0000 7f6167713700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61601030c0 con 0x7f6160102770 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.590+0000 7f61654af700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f6160102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.590+0000 7f61654af700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f6160102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52936/0 (socket says 172.21.15.2:52936) 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.590+0000 7f61654af700 1 -- 172.21.15.2:0/3876771123 learned_addr learned my addr 172.21.15.2:0/3876771123 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.590+0000 7f61654af700 1 -- 172.21.15.2:0/3876771123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6160103200 con 0x7f6160102770 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.591+0000 7f61654af700 1 --2- 172.21.15.2:0/3876771123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f6160102b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61500098e0 tx=0x7f6150009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f061fc05204d3127 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.591+0000 7f6157fff700 1 -- 172.21.15.2:0/3876771123 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f615000b860 con 0x7f6160102770 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.591+0000 7f6157fff700 1 -- 172.21.15.2:0/3876771123 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f615000be90 con 0x7f6160102770 2024-02-16T22:12:49.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.591+0000 7f6157fff700 1 -- 172.21.15.2:0/3876771123 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61500103c0 con 0x7f6160102770 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.591+0000 7f6167713700 1 -- 172.21.15.2:0/3876771123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 msgr2=0x7f6160102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.591+0000 7f6167713700 1 --2- 172.21.15.2:0/3876771123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f6160102b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61500098e0 tx=0x7f6150009250 comp rx=0 tx=0).stop 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.591+0000 7f6167713700 1 -- 172.21.15.2:0/3876771123 shutdown_connections 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.591+0000 7f6167713700 1 --2- 172.21.15.2:0/3876771123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f6160102b80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.591+0000 7f6167713700 1 -- 172.21.15.2:0/3876771123 >> 172.21.15.2:0/3876771123 conn(0x7f61600fded0 msgr2=0x7f6160100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.592+0000 7f6167713700 1 -- 172.21.15.2:0/3876771123 shutdown_connections 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.592+0000 7f6167713700 1 -- 172.21.15.2:0/3876771123 wait complete. 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.592+0000 7f6167713700 1 Processor -- start 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.592+0000 7f6167713700 1 -- start start 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.592+0000 7f6167713700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f61601115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.592+0000 7f6167713700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6160111af0 con 0x7f6160102770 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.592+0000 7f61654af700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f61601115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.592+0000 7f61654af700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f61601115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52940/0 (socket says 172.21.15.2:52940) 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.592+0000 7f61654af700 1 -- 172.21.15.2:0/2965955457 learned_addr learned my addr 172.21.15.2:0/2965955457 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:49.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.593+0000 7f61654af700 1 -- 172.21.15.2:0/2965955457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61500094f0 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.593+0000 7f61654af700 1 --2- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f61601115b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6150011680 tx=0x7f6150009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.593+0000 7f6155ffb700 1 -- 172.21.15.2:0/2965955457 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6150010890 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.593+0000 7f6155ffb700 1 -- 172.21.15.2:0/2965955457 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6150010ec0 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.593+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6160111cf0 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.593+0000 7f6155ffb700 1 -- 172.21.15.2:0/2965955457 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61500243f0 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.593+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6160112990 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.594+0000 7f6155ffb700 1 -- 172.21.15.2:0/2965955457 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f615002d3f0 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.594+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616005f5c0 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.594+0000 7f6155ffb700 1 --2- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f614c036790 0x7f614c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.594+0000 7f6155ffb700 1 -- 172.21.15.2:0/2965955457 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f615004a230 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.595+0000 7f6164cae700 1 --2- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f614c036790 0x7f614c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.595+0000 7f6164cae700 1 --2- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f614c036790 0x7f614c038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f615c009990 tx=0x7f615c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.599+0000 7f6155ffb700 1 -- 172.21.15.2:0/2965955457 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f61500109f0 con 0x7f6160102770 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.803+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f61601001b0 con 0x7f614c036790 2024-02-16T22:12:49.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.807+0000 7f6155ffb700 1 -- 172.21.15.2:0/2965955457 <== mgr.14120 v2:172.21.15.2:6800/673699505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f61601001b0 con 0x7f614c036790 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f614c036790 msgr2=0x7f614c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 --2- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f614c036790 0x7f614c038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f615c009990 tx=0x7f615c006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 msgr2=0x7f61601115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 --2- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f61601115b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6150011680 tx=0x7f6150009250 comp rx=0 tx=0).stop 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 shutdown_connections 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 --2- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f614c036790 0x7f614c038c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 --2- 172.21.15.2:0/2965955457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6160102770 0x7f61601115b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 >> 172.21.15.2:0/2965955457 conn(0x7f61600fded0 msgr2=0x7f61600ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 shutdown_connections 2024-02-16T22:12:49.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:48.809+0000 7f6167713700 1 -- 172.21.15.2:0/2965955457 wait complete. 2024-02-16T22:12:49.947 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:49 smithi002 ceph-mon[29154]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:12:49.947 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:49 smithi002 ceph-mon[29154]: Saving service alertmanager spec with placement count:1 2024-02-16T22:12:49.947 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:49 smithi002 ceph-mon[29154]: from='mgr.14120 172.21.15.2:0/3481344460' entity='mgr.smithi002.resmjv' 2024-02-16T22:12:50.792 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.971+0000 7fb795a6a700 1 Processor -- start 2024-02-16T22:12:50.792 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.971+0000 7fb795a6a700 1 -- start start 2024-02-16T22:12:50.792 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.971+0000 7fb795a6a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7900771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:50.792 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.971+0000 7fb795a6a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7900790f0 con 0x7fb790078ce0 2024-02-16T22:12:50.792 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.971+0000 7fb78f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7900771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.971+0000 7fb78f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7900771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52944/0 (socket says 172.21.15.2:52944) 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.971+0000 7fb78f7fe700 1 -- 172.21.15.2:0/958409104 learned_addr learned my addr 172.21.15.2:0/958409104 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.971+0000 7fb78f7fe700 1 -- 172.21.15.2:0/958409104 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb790077730 con 0x7fb790078ce0 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.972+0000 7fb78f7fe700 1 --2- 172.21.15.2:0/958409104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7900771f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb7800098e0 tx=0x7fb780009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12364402087d02e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.972+0000 7fb78e7fc700 1 -- 172.21.15.2:0/958409104 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb78000b860 con 0x7fb790078ce0 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.972+0000 7fb78e7fc700 1 -- 172.21.15.2:0/958409104 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb78000be90 con 0x7fb790078ce0 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.972+0000 7fb78e7fc700 1 -- 172.21.15.2:0/958409104 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7800103c0 con 0x7fb790078ce0 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.972+0000 7fb795a6a700 1 -- 172.21.15.2:0/958409104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 msgr2=0x7fb7900771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.972+0000 7fb795a6a700 1 --2- 172.21.15.2:0/958409104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7900771f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb7800098e0 tx=0x7fb780009250 comp rx=0 tx=0).stop 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.972+0000 7fb795a6a700 1 -- 172.21.15.2:0/958409104 shutdown_connections 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.972+0000 7fb795a6a700 1 --2- 172.21.15.2:0/958409104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7900771f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.972+0000 7fb795a6a700 1 -- 172.21.15.2:0/958409104 >> 172.21.15.2:0/958409104 conn(0x7fb7900fded0 msgr2=0x7fb790100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.973+0000 7fb795a6a700 1 -- 172.21.15.2:0/958409104 shutdown_connections 2024-02-16T22:12:50.793 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.973+0000 7fb795a6a700 1 -- 172.21.15.2:0/958409104 wait complete. 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.973+0000 7fb795a6a700 1 Processor -- start 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.973+0000 7fb795a6a700 1 -- start start 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.973+0000 7fb795a6a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7901137d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.973+0000 7fb795a6a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb790113d10 con 0x7fb790078ce0 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb78f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7901137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb78f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7901137d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52948/0 (socket says 172.21.15.2:52948) 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb78f7fe700 1 -- 172.21.15.2:0/2862629365 learned_addr learned my addr 172.21.15.2:0/2862629365 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb78f7fe700 1 -- 172.21.15.2:0/2862629365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7800094f0 con 0x7fb790078ce0 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb78f7fe700 1 --2- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7901137d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb7800094c0 tx=0x7fb780009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb777fff700 1 -- 172.21.15.2:0/2862629365 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb780010890 con 0x7fb790078ce0 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb777fff700 1 -- 172.21.15.2:0/2862629365 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb780010ec0 con 0x7fb790078ce0 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb790113f10 con 0x7fb790078ce0 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb777fff700 1 -- 172.21.15.2:0/2862629365 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7800243f0 con 0x7fb790078ce0 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.974+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb790114bb0 con 0x7fb790078ce0 2024-02-16T22:12:50.794 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.975+0000 7fb777fff700 1 -- 172.21.15.2:0/2862629365 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb78001a7e0 con 0x7fb790078ce0 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.975+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79005f5c0 con 0x7fb790078ce0 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.975+0000 7fb777fff700 1 --2- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb7780367e0 0x7fb778038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.975+0000 7fb777fff700 1 -- 172.21.15.2:0/2862629365 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb78004a230 con 0x7fb790078ce0 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.975+0000 7fb78effd700 1 --2- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb7780367e0 0x7fb778038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.976+0000 7fb78effd700 1 --2- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb7780367e0 0x7fb778038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb784009990 tx=0x7fb784006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:49.980+0000 7fb777fff700 1 -- 172.21.15.2:0/2862629365 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb78002d3f0 con 0x7fb790078ce0 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.171+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb79004bed0 con 0x7fb790078ce0 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.174+0000 7fb777fff700 1 -- 172.21.15.2:0/2862629365 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb78002b2f0 con 0x7fb790078ce0 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb7780367e0 msgr2=0x7fb778038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 --2- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb7780367e0 0x7fb778038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb784009990 tx=0x7fb784006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 msgr2=0x7fb7901137d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 --2- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7901137d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb7800094c0 tx=0x7fb780009250 comp rx=0 tx=0).stop 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 shutdown_connections 2024-02-16T22:12:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 --2- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fb7780367e0 0x7fb778038c90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:50.796 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 --2- 172.21.15.2:0/2862629365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb790078ce0 0x7fb7901137d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:50.796 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 >> 172.21.15.2:0/2862629365 conn(0x7fb7900fded0 msgr2=0x7fb7900ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:50.796 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 shutdown_connections 2024-02-16T22:12:50.796 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:50.176+0000 7fb795a6a700 1 -- 172.21.15.2:0/2862629365 wait complete. 2024-02-16T22:12:51.070 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:50 smithi002 ceph-mon[29154]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:12:51.070 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:50 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2862629365' entity='client.admin' 2024-02-16T22:12:51.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.174+0000 7f344dadb700 1 Processor -- start 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.174+0000 7f344dadb700 1 -- start start 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.174+0000 7f344dadb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448102b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.174+0000 7f344dadb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34481030d0 con 0x7f3448102780 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.174+0000 7f34477fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.174+0000 7f34477fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50756/0 (socket says 172.21.15.2:50756) 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.174+0000 7f34477fe700 1 -- 172.21.15.2:0/85624390 learned_addr learned my addr 172.21.15.2:0/85624390 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.174+0000 7f34477fe700 1 -- 172.21.15.2:0/85624390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3448103210 con 0x7f3448102780 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f34477fe700 1 --2- 172.21.15.2:0/85624390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448102b90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f34380098e0 tx=0x7f3438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b7ceb773ec7235 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f34467fc700 1 -- 172.21.15.2:0/85624390 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f343800b860 con 0x7f3448102780 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f34467fc700 1 -- 172.21.15.2:0/85624390 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f343800be90 con 0x7f3448102780 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f34467fc700 1 -- 172.21.15.2:0/85624390 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34380103c0 con 0x7f3448102780 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f344dadb700 1 -- 172.21.15.2:0/85624390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 msgr2=0x7f3448102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f344dadb700 1 --2- 172.21.15.2:0/85624390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448102b90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f34380098e0 tx=0x7f3438009250 comp rx=0 tx=0).stop 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f344dadb700 1 -- 172.21.15.2:0/85624390 shutdown_connections 2024-02-16T22:12:51.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f344dadb700 1 --2- 172.21.15.2:0/85624390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448102b90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f344dadb700 1 -- 172.21.15.2:0/85624390 >> 172.21.15.2:0/85624390 conn(0x7f34480fdf00 msgr2=0x7f3448100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f344dadb700 1 -- 172.21.15.2:0/85624390 shutdown_connections 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.175+0000 7f344dadb700 1 -- 172.21.15.2:0/85624390 wait complete. 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.176+0000 7f344dadb700 1 Processor -- start 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.176+0000 7f344dadb700 1 -- start start 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.176+0000 7f344dadb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.176+0000 7f344dadb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3448111ad0 con 0x7f3448102780 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.176+0000 7f34477fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.176+0000 7f34477fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50772/0 (socket says 172.21.15.2:50772) 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.176+0000 7f34477fe700 1 -- 172.21.15.2:0/558498761 learned_addr learned my addr 172.21.15.2:0/558498761 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.177+0000 7f34477fe700 1 -- 172.21.15.2:0/558498761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34380094f0 con 0x7f3448102780 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.177+0000 7f34477fe700 1 --2- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448111590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f34380090b0 tx=0x7f3438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.177+0000 7f342ffff700 1 -- 172.21.15.2:0/558498761 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3438010840 con 0x7f3448102780 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.177+0000 7f342ffff700 1 -- 172.21.15.2:0/558498761 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3438010e70 con 0x7f3448102780 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.177+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3448111cd0 con 0x7f3448102780 2024-02-16T22:12:51.991 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.177+0000 7f342ffff700 1 -- 172.21.15.2:0/558498761 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3438024e40 con 0x7f3448102780 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.177+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3448112970 con 0x7f3448102780 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.178+0000 7f342ffff700 1 -- 172.21.15.2:0/558498761 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f343801a530 con 0x7f3448102780 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.178+0000 7f342ffff700 1 --2- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f3430036740 0x7f3430038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.178+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344805f5c0 con 0x7f3448102780 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.178+0000 7f342ffff700 1 -- 172.21.15.2:0/558498761 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f343804a810 con 0x7f3448102780 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.178+0000 7f3446ffd700 1 --2- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f3430036740 0x7f3430038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.179+0000 7f3446ffd700 1 --2- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f3430036740 0x7f3430038bf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f343c009990 tx=0x7f343c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.183+0000 7f342ffff700 1 -- 172.21.15.2:0/558498761 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3438018940 con 0x7f3448102780 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.370+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f344804bed0 con 0x7f3448102780 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.374+0000 7f342ffff700 1 -- 172.21.15.2:0/558498761 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f3438048020 con 0x7f3448102780 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f3430036740 msgr2=0x7f3430038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 --2- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f3430036740 0x7f3430038bf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f343c009990 tx=0x7f343c006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 msgr2=0x7f3448111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:51.992 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 --2- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448111590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f34380090b0 tx=0x7f3438009250 comp rx=0 tx=0).stop 2024-02-16T22:12:51.993 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 shutdown_connections 2024-02-16T22:12:51.993 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 --2- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f3430036740 0x7f3430038bf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:51.993 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 --2- 172.21.15.2:0/558498761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3448102780 0x7f3448111590 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:51.993 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 >> 172.21.15.2:0/558498761 conn(0x7f34480fdf00 msgr2=0x7f34480fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:51.993 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 shutdown_connections 2024-02-16T22:12:51.993 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:51.376+0000 7f344dadb700 1 -- 172.21.15.2:0/558498761 wait complete. 2024-02-16T22:12:51.993 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling the dashboard module... 2024-02-16T22:12:52.678 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:52 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/558498761' entity='client.admin' 2024-02-16T22:12:52.678 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:52 smithi002 ceph-mon[29154]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:12:53.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:53 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1137757174' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-16T22:12:54.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.420+0000 7fa625791700 1 Processor -- start 2024-02-16T22:12:54.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.420+0000 7fa625791700 1 -- start start 2024-02-16T22:12:54.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.420+0000 7fa625791700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa620103180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:54.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.420+0000 7fa625791700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6201036c0 con 0x7fa620102d70 2024-02-16T22:12:54.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.420+0000 7fa61effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa620103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:54.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.420+0000 7fa61effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa620103180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50782/0 (socket says 172.21.15.2:50782) 2024-02-16T22:12:54.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.420+0000 7fa61effd700 1 -- 172.21.15.2:0/3567531699 learned_addr learned my addr 172.21.15.2:0/3567531699 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.420+0000 7fa61effd700 1 -- 172.21.15.2:0/3567531699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620103800 con 0x7fa620102d70 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.420+0000 7fa61effd700 1 --2- 172.21.15.2:0/3567531699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa620103180 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6100098e0 tx=0x7fa610009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d733e31cc23aea9e server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa61dffb700 1 -- 172.21.15.2:0/3567531699 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa61000b8f0 con 0x7fa620102d70 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa61dffb700 1 -- 172.21.15.2:0/3567531699 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa610011420 con 0x7fa620102d70 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa61dffb700 1 -- 172.21.15.2:0/3567531699 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa610010540 con 0x7fa620102d70 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa625791700 1 -- 172.21.15.2:0/3567531699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 msgr2=0x7fa620103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa625791700 1 --2- 172.21.15.2:0/3567531699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa620103180 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6100098e0 tx=0x7fa610009250 comp rx=0 tx=0).stop 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa625791700 1 -- 172.21.15.2:0/3567531699 shutdown_connections 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa625791700 1 --2- 172.21.15.2:0/3567531699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa620103180 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa625791700 1 -- 172.21.15.2:0/3567531699 >> 172.21.15.2:0/3567531699 conn(0x7fa6200fc580 msgr2=0x7fa6200fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa625791700 1 -- 172.21.15.2:0/3567531699 shutdown_connections 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.421+0000 7fa625791700 1 -- 172.21.15.2:0/3567531699 wait complete. 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.422+0000 7fa625791700 1 Processor -- start 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.422+0000 7fa625791700 1 -- start start 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.422+0000 7fa625791700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa6201139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.422+0000 7fa625791700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa620113f30 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.422+0000 7fa61effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa6201139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.422+0000 7fa61effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa6201139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50788/0 (socket says 172.21.15.2:50788) 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.422+0000 7fa61effd700 1 -- 172.21.15.2:0/1137757174 learned_addr learned my addr 172.21.15.2:0/1137757174 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.423+0000 7fa61effd700 1 -- 172.21.15.2:0/1137757174 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6100094f0 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.423+0000 7fa61effd700 1 --2- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa6201139f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa610006b00 tx=0x7fa610009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.423+0000 7fa6077fe700 1 -- 172.21.15.2:0/1137757174 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa610010800 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.423+0000 7fa6077fe700 1 -- 172.21.15.2:0/1137757174 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa610010e30 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.423+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa620114130 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.423+0000 7fa6077fe700 1 -- 172.21.15.2:0/1137757174 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6100243f0 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.423+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa620114dd0 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.424+0000 7fa6077fe700 1 -- 172.21.15.2:0/1137757174 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa61002d3f0 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.424+0000 7fa6077fe700 1 --2- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fa6080367e0 0x7fa608038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.424+0000 7fa6077fe700 1 -- 172.21.15.2:0/1137757174 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa61004a230 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.424+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62005f5c0 con 0x7fa620102d70 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.424+0000 7fa61e7fc700 1 --2- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fa6080367e0 0x7fa608038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:54.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.424+0000 7fa61e7fc700 1 --2- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fa6080367e0 0x7fa608038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa614009990 tx=0x7fa614006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.429+0000 7fa6077fe700 1 -- 172.21.15.2:0/1137757174 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa61002b8e0 con 0x7fa620102d70 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:52.676+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fa620108a10 con 0x7fa620102d70 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.379+0000 7fa6077fe700 1 -- 172.21.15.2:0/1137757174 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fa61001a7a0 con 0x7fa620102d70 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.380+0000 7fa6077fe700 1 -- 172.21.15.2:0/1137757174 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fa6100299d0 con 0x7fa620102d70 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fa6080367e0 msgr2=0x7fa608038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 --2- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fa6080367e0 0x7fa608038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa614009990 tx=0x7fa614006e40 comp rx=0 tx=0).stop 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 msgr2=0x7fa6201139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 --2- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa6201139f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa610006b00 tx=0x7fa610009250 comp rx=0 tx=0).stop 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 shutdown_connections 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 --2- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7fa6080367e0 0x7fa608038c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 --2- 172.21.15.2:0/1137757174 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa620102d70 0x7fa6201139f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 >> 172.21.15.2:0/1137757174 conn(0x7fa6200fc580 msgr2=0x7fa6200fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 shutdown_connections 2024-02-16T22:12:54.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:53.383+0000 7fa625791700 1 -- 172.21.15.2:0/1137757174 wait complete. 2024-02-16T22:12:54.682 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:54 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1137757174' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-16T22:12:54.683 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:54 smithi002 ceph-mon[29154]: mgrmap e9: smithi002.resmjv(active, since 25s) 2024-02-16T22:12:55.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-02-16T22:12:55.319 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "smithi002.resmjv", 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.503+0000 7f58917d8700 1 Processor -- start 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.503+0000 7f58917d8700 1 -- start start 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.503+0000 7f58917d8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c077210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.503+0000 7f58917d8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c079110 con 0x7f588c078d00 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.504+0000 7f588affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c077210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.504+0000 7f588affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c077210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50804/0 (socket says 172.21.15.2:50804) 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.504+0000 7f588affd700 1 -- 172.21.15.2:0/814149933 learned_addr learned my addr 172.21.15.2:0/814149933 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.504+0000 7f588affd700 1 -- 172.21.15.2:0/814149933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c077750 con 0x7f588c078d00 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.504+0000 7f588affd700 1 --2- 172.21.15.2:0/814149933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c077210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58740098e0 tx=0x7f5874009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9511d5c937904d33 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f5889ffb700 1 -- 172.21.15.2:0/814149933 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f587400b860 con 0x7f588c078d00 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f5889ffb700 1 -- 172.21.15.2:0/814149933 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f587400be90 con 0x7f588c078d00 2024-02-16T22:12:55.320 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f5889ffb700 1 -- 172.21.15.2:0/814149933 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58740103c0 con 0x7f588c078d00 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f58917d8700 1 -- 172.21.15.2:0/814149933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 msgr2=0x7f588c077210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f58917d8700 1 --2- 172.21.15.2:0/814149933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c077210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58740098e0 tx=0x7f5874009250 comp rx=0 tx=0).stop 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f58917d8700 1 -- 172.21.15.2:0/814149933 shutdown_connections 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f58917d8700 1 --2- 172.21.15.2:0/814149933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c077210 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f58917d8700 1 -- 172.21.15.2:0/814149933 >> 172.21.15.2:0/814149933 conn(0x7f588c0fded0 msgr2=0x7f588c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f58917d8700 1 -- 172.21.15.2:0/814149933 shutdown_connections 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.505+0000 7f58917d8700 1 -- 172.21.15.2:0/814149933 wait complete. 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.506+0000 7f58917d8700 1 Processor -- start 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.506+0000 7f58917d8700 1 -- start start 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.506+0000 7f58917d8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.506+0000 7f58917d8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c113cc0 con 0x7f588c078d00 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.506+0000 7f588affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.506+0000 7f588affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50812/0 (socket says 172.21.15.2:50812) 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.506+0000 7f588affd700 1 -- 172.21.15.2:0/1945400678 learned_addr learned my addr 172.21.15.2:0/1945400678 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.507+0000 7f588affd700 1 -- 172.21.15.2:0/1945400678 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58740094f0 con 0x7f588c078d00 2024-02-16T22:12:55.321 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.507+0000 7f588affd700 1 --2- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c113780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5874003080 tx=0x7f5874009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.507+0000 7f58837fe700 1 -- 172.21.15.2:0/1945400678 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5874011920 con 0x7f588c078d00 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.507+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c113ec0 con 0x7f588c078d00 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.507+0000 7f58837fe700 1 -- 172.21.15.2:0/1945400678 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5874010d60 con 0x7f588c078d00 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.507+0000 7f58837fe700 1 -- 172.21.15.2:0/1945400678 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58740243f0 con 0x7f588c078d00 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.507+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c114b60 con 0x7f588c078d00 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.508+0000 7f58837fe700 1 -- 172.21.15.2:0/1945400678 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f587402d3f0 con 0x7f588c078d00 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.508+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c05f5c0 con 0x7f588c078d00 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.508+0000 7f58837fe700 1 --2- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f58780367e0 0x7f5878038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.508+0000 7f58837fe700 1 -- 172.21.15.2:0/1945400678 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f587404a430 con 0x7f588c078d00 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.508+0000 7f588a7fc700 1 -- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f58780367e0 msgr2=0x7f5878038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/673699505 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.508+0000 7f588a7fc700 1 --2- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f58780367e0 0x7f5878038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.513+0000 7f58837fe700 1 -- 172.21.15.2:0/1945400678 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f587402b8e0 con 0x7f588c078d00 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.709+0000 7f588a7fc700 1 -- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f58780367e0 msgr2=0x7f5878038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/673699505 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.709+0000 7f588a7fc700 1 --2- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f58780367e0 0x7f5878038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-16T22:12:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.761+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f588c114ea0 con 0x7f588c078d00 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.761+0000 7f58837fe700 1 -- 172.21.15.2:0/1945400678 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f587402b2f0 con 0x7f588c078d00 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f58780367e0 msgr2=0x7f5878038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 --2- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f58780367e0 0x7f5878038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 msgr2=0x7f588c113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 --2- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c113780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5874003080 tx=0x7f5874009250 comp rx=0 tx=0).stop 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 shutdown_connections 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 --2- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f58780367e0 0x7f5878038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 --2- 172.21.15.2:0/1945400678 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588c078d00 0x7f588c113780 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 >> 172.21.15.2:0/1945400678 conn(0x7f588c0fded0 msgr2=0x7f588c0ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 shutdown_connections 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:54.763+0000 7f58917d8700 1 -- 172.21.15.2:0/1945400678 wait complete. 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2024-02-16T22:12:55.323 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 9... 2024-02-16T22:12:55.547 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:12:55 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1945400678' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-16T22:13:01.123 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: Active manager daemon smithi002.resmjv restarted 2024-02-16T22:13:01.123 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: Activating manager daemon smithi002.resmjv 2024-02-16T22:13:01.123 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: osdmap e3: 0 total, 0 up, 0 in 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: mgrmap e10: smithi002.resmjv(active, starting, since 0.00319217s) 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr metadata", "who": "smithi002.resmjv", "id": "smithi002.resmjv"}]: dispatch 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: Manager daemon smithi002.resmjv is now available 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/mirror_snapshot_schedule"}]: dispatch 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/trash_purge_schedule"}]: dispatch 2024-02-16T22:13:01.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:00 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.788+0000 7f05e6af0700 1 Processor -- start 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.788+0000 7f05e6af0700 1 -- start start 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.789+0000 7f05e6af0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.789+0000 7f05e6af0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e0105960 con 0x7f05e00713e0 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.789+0000 7f05e488c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.789+0000 7f05e488c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50822/0 (socket says 172.21.15.2:50822) 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.789+0000 7f05e488c700 1 -- 172.21.15.2:0/1709667914 learned_addr learned my addr 172.21.15.2:0/1709667914 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.789+0000 7f05e488c700 1 -- 172.21.15.2:0/1709667914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e0105aa0 con 0x7f05e00713e0 2024-02-16T22:13:01.757 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.789+0000 7f05e488c700 1 --2- 172.21.15.2:0/1709667914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0105420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f05d40098e0 tx=0x7f05d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4213fbb353962f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.789+0000 7f05df7fe700 1 -- 172.21.15.2:0/1709667914 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05d400b860 con 0x7f05e00713e0 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.790+0000 7f05df7fe700 1 -- 172.21.15.2:0/1709667914 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05d400be90 con 0x7f05e00713e0 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.790+0000 7f05df7fe700 1 -- 172.21.15.2:0/1709667914 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05d4009e80 con 0x7f05e00713e0 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.790+0000 7f05e6af0700 1 -- 172.21.15.2:0/1709667914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 msgr2=0x7f05e0105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.790+0000 7f05e6af0700 1 --2- 172.21.15.2:0/1709667914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0105420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f05d40098e0 tx=0x7f05d4009250 comp rx=0 tx=0).stop 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.790+0000 7f05e6af0700 1 -- 172.21.15.2:0/1709667914 shutdown_connections 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.790+0000 7f05e6af0700 1 --2- 172.21.15.2:0/1709667914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0105420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.790+0000 7f05e6af0700 1 -- 172.21.15.2:0/1709667914 >> 172.21.15.2:0/1709667914 conn(0x7f05e00fcdd0 msgr2=0x7f05e00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.791+0000 7f05e6af0700 1 -- 172.21.15.2:0/1709667914 shutdown_connections 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.791+0000 7f05e6af0700 1 -- 172.21.15.2:0/1709667914 wait complete. 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.791+0000 7f05e6af0700 1 Processor -- start 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.791+0000 7f05e6af0700 1 -- start start 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.791+0000 7f05e6af0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.791+0000 7f05e6af0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e0101320 con 0x7f05e00713e0 2024-02-16T22:13:01.758 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.792+0000 7f05e488c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.792+0000 7f05e488c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0102ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50830/0 (socket says 172.21.15.2:50830) 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.792+0000 7f05e488c700 1 -- 172.21.15.2:0/3158622948 learned_addr learned my addr 172.21.15.2:0/3158622948 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.792+0000 7f05e488c700 1 -- 172.21.15.2:0/3158622948 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d40094f0 con 0x7f05e00713e0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.792+0000 7f05e488c700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0102ea0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f05d40094c0 tx=0x7f05d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.792+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05d401a7a0 con 0x7f05e00713e0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.792+0000 7f05e6af0700 1 -- 172.21.15.2:0/3158622948 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e0101520 con 0x7f05e00713e0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.792+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05d401add0 con 0x7f05e00713e0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.792+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05d4019d80 con 0x7f05e00713e0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.793+0000 7f05e6af0700 1 -- 172.21.15.2:0/3158622948 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e01019b0 con 0x7f05e00713e0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.793+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f05d402d3f0 con 0x7f05e00713e0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.794+0000 7f05dd7fa700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 0x7f05d0038cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.794+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 --> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05d0039400 con 0x7f05d0036840 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.794+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f05d404a3f0 con 0x7f05e00713e0 2024-02-16T22:13:01.759 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.794+0000 7f05dffff700 1 -- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 msgr2=0x7f05d0038cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/673699505 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.794+0000 7f05dffff700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 0x7f05d0038cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.995+0000 7f05dffff700 1 -- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 msgr2=0x7f05d0038cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/673699505 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:55.995+0000 7f05dffff700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 0x7f05d0038cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:56.396+0000 7f05dffff700 1 -- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 msgr2=0x7f05d0038cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/673699505 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:56.396+0000 7f05dffff700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 0x7f05d0038cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:57.197+0000 7f05dffff700 1 -- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 msgr2=0x7f05d0038cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/673699505 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:57.197+0000 7f05dffff700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 0x7f05d0038cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:58.798+0000 7f05dffff700 1 -- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 msgr2=0x7f05d0038cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/673699505 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:12:58.798+0000 7f05dffff700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 0x7f05d0038cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:00.119+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7f05d4019630 con 0x7f05e00713e0 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:00.119+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 msgr2=0x7f05d0038cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:00.119+0000 7f05dd7fa700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 0x7f05d0038cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:01.760 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.120+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f05d4049a30 con 0x7f05e00713e0 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.120+0000 7f05dd7fa700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f05d0039fe0 0x7f05d003c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.120+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 --> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05d0039400 con 0x7f05d0039fe0 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.121+0000 7f05dffff700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f05d0039fe0 0x7f05d003c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.122+0000 7f05dffff700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f05d0039fe0 0x7f05d003c3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f05cc0039d0 tx=0x7f05cc0092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.122+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 <== mgr.14162 v2:172.21.15.2:6800/2481260931 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f05d0039400 con 0x7f05d0039fe0 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.128+0000 7f05e6af0700 1 -- 172.21.15.2:0/3158622948 --> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f05e005f5c0 con 0x7f05d0039fe0 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.128+0000 7f05dd7fa700 1 -- 172.21.15.2:0/3158622948 <== mgr.14162 v2:172.21.15.2:6800/2481260931 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f05e005f5c0 con 0x7f05d0039fe0 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 -- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f05d0039fe0 msgr2=0x7f05d003c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f05d0039fe0 0x7f05d003c3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f05cc0039d0 tx=0x7f05cc0092f0 comp rx=0 tx=0).stop 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 -- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 msgr2=0x7f05e0102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0102ea0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f05d40094c0 tx=0x7f05d4009250 comp rx=0 tx=0).stop 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 -- 172.21.15.2:0/3158622948 shutdown_connections 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f05d0039fe0 0x7f05d003c3c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05e00713e0 0x7f05e0102ea0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 --2- 172.21.15.2:0/3158622948 >> [v2:172.21.15.2:6800/673699505,v1:172.21.15.2:6801/673699505] conn(0x7f05d0036840 0x7f05d0038cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:01.761 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 -- 172.21.15.2:0/3158622948 >> 172.21.15.2:0/3158622948 conn(0x7f05e00fcdd0 msgr2=0x7f05e00fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:01.762 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 -- 172.21.15.2:0/3158622948 shutdown_connections 2024-02-16T22:13:01.762 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:01.129+0000 7f05e6af0700 1 -- 172.21.15.2:0/3158622948 wait complete. 2024-02-16T22:13:01.762 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 9 is available 2024-02-16T22:13:01.762 INFO:teuthology.orchestra.run.smithi002.stdout:Generating a dashboard self-signed certificate... 2024-02-16T22:13:02.396 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:02 smithi002 ceph-mon[29154]: [16/Feb/2024:22:13:00] ENGINE Bus STARTING 2024-02-16T22:13:02.396 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:02 smithi002 ceph-mon[29154]: [16/Feb/2024:22:13:00] ENGINE Serving on https://172.21.15.2:7150 2024-02-16T22:13:02.396 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:02 smithi002 ceph-mon[29154]: [16/Feb/2024:22:13:00] ENGINE Bus STARTED 2024-02-16T22:13:02.396 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:02 smithi002 ceph-mon[29154]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-16T22:13:02.396 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:02 smithi002 ceph-mon[29154]: mgrmap e11: smithi002.resmjv(active, since 1.00574s) 2024-02-16T22:13:02.396 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:02 smithi002 ceph-mon[29154]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-16T22:13:02.397 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:02 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:03.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-16T22:13:03.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.180+0000 7f3c85009700 1 Processor -- start 2024-02-16T22:13:03.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.180+0000 7f3c85009700 1 -- start start 2024-02-16T22:13:03.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.180+0000 7f3c85009700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c80101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.180+0000 7f3c85009700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c80101fc0 con 0x7f3c80101670 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.180+0000 7f3c7ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c80101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.180+0000 7f3c7ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c80101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60662/0 (socket says 172.21.15.2:60662) 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.180+0000 7f3c7ed9d700 1 -- 172.21.15.2:0/1869298118 learned_addr learned my addr 172.21.15.2:0/1869298118 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.180+0000 7f3c7ed9d700 1 -- 172.21.15.2:0/1869298118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80102100 con 0x7f3c80101670 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.181+0000 7f3c7ed9d700 1 --2- 172.21.15.2:0/1869298118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c80101a80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3c700098e0 tx=0x7f3c70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d973577e6120759b server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.181+0000 7f3c7dd9b700 1 -- 172.21.15.2:0/1869298118 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7000b860 con 0x7f3c80101670 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.181+0000 7f3c7dd9b700 1 -- 172.21.15.2:0/1869298118 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c7000be90 con 0x7f3c80101670 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.181+0000 7f3c7dd9b700 1 -- 172.21.15.2:0/1869298118 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c700103c0 con 0x7f3c80101670 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.181+0000 7f3c85009700 1 -- 172.21.15.2:0/1869298118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 msgr2=0x7f3c80101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.181+0000 7f3c85009700 1 --2- 172.21.15.2:0/1869298118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c80101a80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3c700098e0 tx=0x7f3c70009250 comp rx=0 tx=0).stop 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.181+0000 7f3c85009700 1 -- 172.21.15.2:0/1869298118 shutdown_connections 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.181+0000 7f3c85009700 1 --2- 172.21.15.2:0/1869298118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c80101a80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.181+0000 7f3c85009700 1 -- 172.21.15.2:0/1869298118 >> 172.21.15.2:0/1869298118 conn(0x7f3c800fcdd0 msgr2=0x7f3c800ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:03.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.182+0000 7f3c85009700 1 -- 172.21.15.2:0/1869298118 shutdown_connections 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.182+0000 7f3c85009700 1 -- 172.21.15.2:0/1869298118 wait complete. 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.182+0000 7f3c85009700 1 Processor -- start 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.182+0000 7f3c85009700 1 -- start start 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c85009700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c8010f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c85009700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8010f860 con 0x7f3c80101670 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c7ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c8010f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c7ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c8010f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60678/0 (socket says 172.21.15.2:60678) 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c7ed9d700 1 -- 172.21.15.2:0/2617242683 learned_addr learned my addr 172.21.15.2:0/2617242683 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c7ed9d700 1 -- 172.21.15.2:0/2617242683 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c700094f0 con 0x7f3c80101670 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c7ed9d700 1 --2- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c8010f320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3c70011650 tx=0x7f3c70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c677fe700 1 -- 172.21.15.2:0/2617242683 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c70010890 con 0x7f3c80101670 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8010fa60 con 0x7f3c80101670 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c677fe700 1 -- 172.21.15.2:0/2617242683 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c70010ec0 con 0x7f3c80101670 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c677fe700 1 -- 172.21.15.2:0/2617242683 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7001a430 con 0x7f3c80101670 2024-02-16T22:13:03.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.183+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c80110700 con 0x7f3c80101670 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.184+0000 7f3c677fe700 1 -- 172.21.15.2:0/2617242683 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f3c7001a660 con 0x7f3c80101670 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.185+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8005f5c0 con 0x7f3c80101670 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.185+0000 7f3c677fe700 1 --2- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3c68038950 0x7f3c6803ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.185+0000 7f3c677fe700 1 -- 172.21.15.2:0/2617242683 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3c7004a340 con 0x7f3c80101670 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.185+0000 7f3c7e59c700 1 --2- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3c68038950 0x7f3c6803ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.185+0000 7f3c7e59c700 1 --2- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3c68038950 0x7f3c6803ae00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009990 tx=0x7f3c74006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.193+0000 7f3c677fe700 1 -- 172.21.15.2:0/2617242683 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3c70018940 con 0x7f3c80101670 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.394+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 --> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c80111280 con 0x7f3c68038950 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.595+0000 7f3c677fe700 1 -- 172.21.15.2:0/2617242683 <== mgr.14162 v2:172.21.15.2:6800/2481260931 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3c80111280 con 0x7f3c68038950 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3c68038950 msgr2=0x7f3c6803ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 --2- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3c68038950 0x7f3c6803ae00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009990 tx=0x7f3c74006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 msgr2=0x7f3c8010f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 --2- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c8010f320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3c70011650 tx=0x7f3c70009250 comp rx=0 tx=0).stop 2024-02-16T22:13:03.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 shutdown_connections 2024-02-16T22:13:03.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 --2- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3c68038950 0x7f3c6803ae00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:03.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 --2- 172.21.15.2:0/2617242683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c80101670 0x7f3c8010f320 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:03.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 >> 172.21.15.2:0/2617242683 conn(0x7f3c800fcdd0 msgr2=0x7f3c800ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:03.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 shutdown_connections 2024-02-16T22:13:03.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:02.597+0000 7f3c85009700 1 -- 172.21.15.2:0/2617242683 wait complete. 2024-02-16T22:13:03.130 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial admin user... 2024-02-16T22:13:03.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:03 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:03.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:03 smithi002 ceph-mon[29154]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:13:03.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:03 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:03.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:03 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:03.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:03 smithi002 ceph-mon[29154]: mgrmap e12: smithi002.resmjv(active, since 3s) 2024-02-16T22:13:04.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$xB4uimMqnlNwJv9RqMEecOAdp87i2bURF35WD7z8vgpWRViYPTKdS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708121583, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-16T22:13:04.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.518+0000 7f7042131700 1 Processor -- start 2024-02-16T22:13:04.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.518+0000 7f7042131700 1 -- start start 2024-02-16T22:13:04.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.518+0000 7f7042131700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:04.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.518+0000 7f7042131700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c105960 con 0x7f703c076b80 2024-02-16T22:13:04.545 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.518+0000 7f703b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.518+0000 7f703b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60690/0 (socket says 172.21.15.2:60690) 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.518+0000 7f703b7fe700 1 -- 172.21.15.2:0/3334891981 learned_addr learned my addr 172.21.15.2:0/3334891981 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.518+0000 7f703b7fe700 1 -- 172.21.15.2:0/3334891981 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c105aa0 con 0x7f703c076b80 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f703b7fe700 1 --2- 172.21.15.2:0/3334891981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c105420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f702c0098e0 tx=0x7f702c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa633f582a1e6c0e server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f703a7fc700 1 -- 172.21.15.2:0/3334891981 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f702c00b860 con 0x7f703c076b80 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f703a7fc700 1 -- 172.21.15.2:0/3334891981 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f702c00be90 con 0x7f703c076b80 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f703a7fc700 1 -- 172.21.15.2:0/3334891981 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f702c0103c0 con 0x7f703c076b80 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f7042131700 1 -- 172.21.15.2:0/3334891981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 msgr2=0x7f703c105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f7042131700 1 --2- 172.21.15.2:0/3334891981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c105420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f702c0098e0 tx=0x7f702c009250 comp rx=0 tx=0).stop 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f7042131700 1 -- 172.21.15.2:0/3334891981 shutdown_connections 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f7042131700 1 --2- 172.21.15.2:0/3334891981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c105420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f7042131700 1 -- 172.21.15.2:0/3334891981 >> 172.21.15.2:0/3334891981 conn(0x7f703c0fcdd0 msgr2=0x7f703c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.519+0000 7f7042131700 1 -- 172.21.15.2:0/3334891981 shutdown_connections 2024-02-16T22:13:04.546 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.520+0000 7f7042131700 1 -- 172.21.15.2:0/3334891981 wait complete. 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.520+0000 7f7042131700 1 Processor -- start 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.520+0000 7f7042131700 1 -- start start 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.520+0000 7f7042131700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c1118e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.520+0000 7f7042131700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c111e20 con 0x7f703c076b80 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.520+0000 7f703b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c1118e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.520+0000 7f703b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c1118e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60694/0 (socket says 172.21.15.2:60694) 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.520+0000 7f703b7fe700 1 -- 172.21.15.2:0/1073527883 learned_addr learned my addr 172.21.15.2:0/1073527883 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.521+0000 7f703b7fe700 1 -- 172.21.15.2:0/1073527883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c0094f0 con 0x7f703c076b80 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.521+0000 7f703b7fe700 1 --2- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c1118e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f702c0090b0 tx=0x7f702c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.521+0000 7f7023fff700 1 -- 172.21.15.2:0/1073527883 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f702c010890 con 0x7f703c076b80 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.521+0000 7f7023fff700 1 -- 172.21.15.2:0/1073527883 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f702c010ec0 con 0x7f703c076b80 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.521+0000 7f7023fff700 1 -- 172.21.15.2:0/1073527883 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f702c024e70 con 0x7f703c076b80 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.521+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c112020 con 0x7f703c076b80 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.521+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c112cc0 con 0x7f703c076b80 2024-02-16T22:13:04.547 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.522+0000 7f7023fff700 1 -- 172.21.15.2:0/1073527883 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f702c024430 con 0x7f703c076b80 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.522+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c05f5c0 con 0x7f703c076b80 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.522+0000 7f7023fff700 1 --2- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7024036830 0x7f7024038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.522+0000 7f7023fff700 1 -- 172.21.15.2:0/1073527883 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f702c04a8f0 con 0x7f703c076b80 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.522+0000 7f703affd700 1 --2- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7024036830 0x7f7024038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.523+0000 7f703affd700 1 --2- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7024036830 0x7f7024038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7030009990 tx=0x7f7030006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.527+0000 7f7023fff700 1 -- 172.21.15.2:0/1073527883 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f702c02c700 con 0x7f703c076b80 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.738+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 --> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f703c05f3e0 con 0x7f7024036830 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.984+0000 7f7023fff700 1 -- 172.21.15.2:0/1073527883 <== mgr.14162 v2:172.21.15.2:6800/2481260931 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f703c05f3e0 con 0x7f7024036830 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7024036830 msgr2=0x7f7024038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 --2- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7024036830 0x7f7024038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7030009990 tx=0x7f7030006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 msgr2=0x7f703c1118e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 --2- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c1118e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f702c0090b0 tx=0x7f702c009250 comp rx=0 tx=0).stop 2024-02-16T22:13:04.548 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 shutdown_connections 2024-02-16T22:13:04.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 --2- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7024036830 0x7f7024038ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:04.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 --2- 172.21.15.2:0/1073527883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f703c076b80 0x7f703c1118e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:04.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 >> 172.21.15.2:0/1073527883 conn(0x7f703c0fcdd0 msgr2=0x7f703c0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:04.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 shutdown_connections 2024-02-16T22:13:04.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:03.986+0000 7f7042131700 1 -- 172.21.15.2:0/1073527883 wait complete. 2024-02-16T22:13:04.549 INFO:teuthology.orchestra.run.smithi002.stdout:Fetching dashboard port number... 2024-02-16T22:13:05.201 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:04 smithi002 ceph-mon[29154]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:13:05.202 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:04 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 8443 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.996+0000 7fe979bd7700 1 Processor -- start 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.996+0000 7fe979bd7700 1 -- start start 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.996+0000 7fe979bd7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974104230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.997+0000 7fe979bd7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe974104770 con 0x7fe974103e20 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.997+0000 7fe9737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974104230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.997+0000 7fe9737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974104230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60696/0 (socket says 172.21.15.2:60696) 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.997+0000 7fe9737fe700 1 -- 172.21.15.2:0/879172116 learned_addr learned my addr 172.21.15.2:0/879172116 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.997+0000 7fe9737fe700 1 -- 172.21.15.2:0/879172116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9741048b0 con 0x7fe974103e20 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.997+0000 7fe9737fe700 1 --2- 172.21.15.2:0/879172116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974104230 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe9640098e0 tx=0x7fe964009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29db70fd646df08 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.997+0000 7fe972ffd700 1 -- 172.21.15.2:0/879172116 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe96400b860 con 0x7fe974103e20 2024-02-16T22:13:05.914 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.997+0000 7fe972ffd700 1 -- 172.21.15.2:0/879172116 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe96400be90 con 0x7fe974103e20 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.997+0000 7fe972ffd700 1 -- 172.21.15.2:0/879172116 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9640103c0 con 0x7fe974103e20 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.998+0000 7fe979bd7700 1 -- 172.21.15.2:0/879172116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 msgr2=0x7fe974104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.998+0000 7fe979bd7700 1 --2- 172.21.15.2:0/879172116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974104230 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe9640098e0 tx=0x7fe964009250 comp rx=0 tx=0).stop 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.998+0000 7fe979bd7700 1 -- 172.21.15.2:0/879172116 shutdown_connections 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.998+0000 7fe979bd7700 1 --2- 172.21.15.2:0/879172116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974104230 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.998+0000 7fe979bd7700 1 -- 172.21.15.2:0/879172116 >> 172.21.15.2:0/879172116 conn(0x7fe9740fc580 msgr2=0x7fe9740fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.998+0000 7fe979bd7700 1 -- 172.21.15.2:0/879172116 shutdown_connections 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.998+0000 7fe979bd7700 1 -- 172.21.15.2:0/879172116 wait complete. 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.998+0000 7fe979bd7700 1 Processor -- start 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe979bd7700 1 -- start start 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe979bd7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974111840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe979bd7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe974111d80 con 0x7fe974103e20 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe9737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974111840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe9737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974111840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60700/0 (socket says 172.21.15.2:60700) 2024-02-16T22:13:05.915 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe9737fe700 1 -- 172.21.15.2:0/423145594 learned_addr learned my addr 172.21.15.2:0/423145594 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe9737fe700 1 -- 172.21.15.2:0/423145594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9640094f0 con 0x7fe974103e20 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe9737fe700 1 --2- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974111840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe964006b00 tx=0x7fe964009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe970ff9700 1 -- 172.21.15.2:0/423145594 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9640109a0 con 0x7fe974103e20 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe970ff9700 1 -- 172.21.15.2:0/423145594 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe96401a430 con 0x7fe974103e20 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe974111f80 con 0x7fe974103e20 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:04.999+0000 7fe970ff9700 1 -- 172.21.15.2:0/423145594 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9640245b0 con 0x7fe974103e20 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.000+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe974112c20 con 0x7fe974103e20 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.000+0000 7fe970ff9700 1 -- 172.21.15.2:0/423145594 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fe9640247e0 con 0x7fe974103e20 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.000+0000 7fe970ff9700 1 --2- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe960036830 0x7fe960038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.000+0000 7fe970ff9700 1 -- 172.21.15.2:0/423145594 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe96404aa70 con 0x7fe974103e20 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.001+0000 7fe96affd700 1 --2- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe960036830 0x7fe960038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.001+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe974112410 con 0x7fe974103e20 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.001+0000 7fe96affd700 1 --2- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe960036830 0x7fe960038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009990 tx=0x7fe95c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:05.916 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.005+0000 7fe970ff9700 1 -- 172.21.15.2:0/423145594 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe964018db0 con 0x7fe974103e20 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.200+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fe97405f5c0 con 0x7fe974103e20 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.200+0000 7fe970ff9700 1 -- 172.21.15.2:0/423145594 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fe964018350 con 0x7fe974103e20 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe960036830 msgr2=0x7fe960038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 --2- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe960036830 0x7fe960038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe95c009990 tx=0x7fe95c006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 msgr2=0x7fe974111840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 --2- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974111840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe964006b00 tx=0x7fe964009250 comp rx=0 tx=0).stop 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 shutdown_connections 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 --2- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe960036830 0x7fe960038ce0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 --2- 172.21.15.2:0/423145594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe974103e20 0x7fe974111840 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 >> 172.21.15.2:0/423145594 conn(0x7fe9740fc580 msgr2=0x7fe9740fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 shutdown_connections 2024-02-16T22:13:05.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:05.202+0000 7fe979bd7700 1 -- 172.21.15.2:0/423145594 wait complete. 2024-02-16T22:13:05.921 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-16T22:13:05.921 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-02-16T22:13:05.928 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-16T22:13:05.928 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-02-16T22:13:05.928 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-02-16T22:13:05.928 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-16T22:13:05.930 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph Dashboard is now available at: 2024-02-16T22:13:05.930 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:05.930 INFO:teuthology.orchestra.run.smithi002.stdout: URL: https://smithi002.front.sepia.ceph.com:8443/ 2024-02-16T22:13:05.930 INFO:teuthology.orchestra.run.smithi002.stdout: User: admin 2024-02-16T22:13:05.930 INFO:teuthology.orchestra.run.smithi002.stdout: Password: aqr2qojo4s 2024-02-16T22:13:05.931 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:05.931 INFO:teuthology.orchestra.run.smithi002.stdout:Saving cluster configuration to /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config directory 2024-02-16T22:13:05.931 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling autotune for osd_memory_target 2024-02-16T22:13:06.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:05 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/423145594' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.503+0000 7f5af733c700 1 Processor -- start 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5af733c700 1 -- start start 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5af733c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af0105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5af733c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af01058c0 con 0x7f5af0071480 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5af50d8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af0105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5af50d8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af0105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60712/0 (socket says 172.21.15.2:60712) 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5af50d8700 1 -- 172.21.15.2:0/1495209204 learned_addr learned my addr 172.21.15.2:0/1495209204 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5af50d8700 1 -- 172.21.15.2:0/1495209204 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af0105a00 con 0x7f5af0071480 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5af50d8700 1 --2- 172.21.15.2:0/1495209204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af0105380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0098e0 tx=0x7f5aec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2732515c1595cc server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5ae7fff700 1 -- 172.21.15.2:0/1495209204 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aec00b860 con 0x7f5af0071480 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5ae7fff700 1 -- 172.21.15.2:0/1495209204 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5aec00be90 con 0x7f5af0071480 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.504+0000 7f5ae7fff700 1 -- 172.21.15.2:0/1495209204 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aec0103c0 con 0x7f5af0071480 2024-02-16T22:13:07.338 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.505+0000 7f5af733c700 1 -- 172.21.15.2:0/1495209204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 msgr2=0x7f5af0105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.505+0000 7f5af733c700 1 --2- 172.21.15.2:0/1495209204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af0105380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0098e0 tx=0x7f5aec009250 comp rx=0 tx=0).stop 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.505+0000 7f5af733c700 1 -- 172.21.15.2:0/1495209204 shutdown_connections 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.505+0000 7f5af733c700 1 --2- 172.21.15.2:0/1495209204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af0105380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.505+0000 7f5af733c700 1 -- 172.21.15.2:0/1495209204 >> 172.21.15.2:0/1495209204 conn(0x7f5af00fcdd0 msgr2=0x7f5af00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.505+0000 7f5af733c700 1 -- 172.21.15.2:0/1495209204 shutdown_connections 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.505+0000 7f5af733c700 1 -- 172.21.15.2:0/1495209204 wait complete. 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.506+0000 7f5af733c700 1 Processor -- start 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.506+0000 7f5af733c700 1 -- start start 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.506+0000 7f5af733c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af01115e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.506+0000 7f5af733c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0111b20 con 0x7f5af0071480 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.506+0000 7f5af50d8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af01115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.506+0000 7f5af50d8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af01115e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60728/0 (socket says 172.21.15.2:60728) 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.506+0000 7f5af50d8700 1 -- 172.21.15.2:0/2202761215 learned_addr learned my addr 172.21.15.2:0/2202761215 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.506+0000 7f5af50d8700 1 -- 172.21.15.2:0/2202761215 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0094f0 con 0x7f5af0071480 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.506+0000 7f5af50d8700 1 --2- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af01115e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006d40 tx=0x7f5aec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:07.339 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.507+0000 7f5ae5ffb700 1 -- 172.21.15.2:0/2202761215 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aec010840 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.507+0000 7f5ae5ffb700 1 -- 172.21.15.2:0/2202761215 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5aec010e70 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.507+0000 7f5ae5ffb700 1 -- 172.21.15.2:0/2202761215 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aec024ea0 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.507+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af0111d20 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.507+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af01129c0 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.507+0000 7f5ae5ffb700 1 -- 172.21.15.2:0/2202761215 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f5aec0109a0 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.508+0000 7f5ae5ffb700 1 --2- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5adc0367e0 0x7f5adc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.508+0000 7f5ae5ffb700 1 -- 172.21.15.2:0/2202761215 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5aec04a7f0 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.508+0000 7f5af48d7700 1 --2- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5adc0367e0 0x7f5adc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.508+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af01121b0 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.508+0000 7f5af48d7700 1 --2- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5adc0367e0 0x7f5adc038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009990 tx=0x7f5ae0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.512+0000 7f5ae5ffb700 1 -- 172.21.15.2:0/2202761215 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5aec02e440 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.703+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f5af0106810 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.703+0000 7f5ae5ffb700 1 -- 172.21.15.2:0/2202761215 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5aec02c7b0 con 0x7f5af0071480 2024-02-16T22:13:07.340 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5adc0367e0 msgr2=0x7f5adc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:07.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 --2- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5adc0367e0 0x7f5adc038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009990 tx=0x7f5ae0006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:07.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 msgr2=0x7f5af01115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:07.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 --2- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af01115e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006d40 tx=0x7f5aec009250 comp rx=0 tx=0).stop 2024-02-16T22:13:07.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 shutdown_connections 2024-02-16T22:13:07.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 --2- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5adc0367e0 0x7f5adc038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:07.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 --2- 172.21.15.2:0/2202761215 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0071480 0x7f5af01115e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:07.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 >> 172.21.15.2:0/2202761215 conn(0x7f5af00fcdd0 msgr2=0x7f5af00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:07.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 shutdown_connections 2024-02-16T22:13:07.341 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:06.705+0000 7f5af733c700 1 -- 172.21.15.2:0/2202761215 wait complete. 2024-02-16T22:13:08.377 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:07 smithi002 ceph-mon[29154]: mgrmap e13: smithi002.resmjv(active, since 6s) 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.754+0000 7f35a0ba0700 1 Processor -- start 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.755+0000 7f35a0ba0700 1 -- start start 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.755+0000 7f35a0ba0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c1077c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.755+0000 7f35a0ba0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1031b0 con 0x7f359c076be0 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.755+0000 7f359a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c1077c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.755+0000 7f359a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c1077c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60738/0 (socket says 172.21.15.2:60738) 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.755+0000 7f359a59c700 1 -- 172.21.15.2:0/155270684 learned_addr learned my addr 172.21.15.2:0/155270684 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.755+0000 7f359a59c700 1 -- 172.21.15.2:0/155270684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c107d00 con 0x7f359c076be0 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f359a59c700 1 --2- 172.21.15.2:0/155270684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c1077c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f358c0098e0 tx=0x7f358c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7763e863a2ce1b9e server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f359959a700 1 -- 172.21.15.2:0/155270684 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f358c00b860 con 0x7f359c076be0 2024-02-16T22:13:08.661 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f359959a700 1 -- 172.21.15.2:0/155270684 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f358c00be90 con 0x7f359c076be0 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f359959a700 1 -- 172.21.15.2:0/155270684 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f358c0103c0 con 0x7f359c076be0 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f35a0ba0700 1 -- 172.21.15.2:0/155270684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 msgr2=0x7f359c1077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f35a0ba0700 1 --2- 172.21.15.2:0/155270684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c1077c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f358c0098e0 tx=0x7f358c009250 comp rx=0 tx=0).stop 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f35a0ba0700 1 -- 172.21.15.2:0/155270684 shutdown_connections 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f35a0ba0700 1 --2- 172.21.15.2:0/155270684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c1077c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f35a0ba0700 1 -- 172.21.15.2:0/155270684 >> 172.21.15.2:0/155270684 conn(0x7f359c0fcdd0 msgr2=0x7f359c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f35a0ba0700 1 -- 172.21.15.2:0/155270684 shutdown_connections 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.756+0000 7f35a0ba0700 1 -- 172.21.15.2:0/155270684 wait complete. 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.757+0000 7f35a0ba0700 1 Processor -- start 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.757+0000 7f35a0ba0700 1 -- start start 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.757+0000 7f35a0ba0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.757+0000 7f35a0ba0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c113fb0 con 0x7f359c076be0 2024-02-16T22:13:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.757+0000 7f359a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.757+0000 7f359a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60744/0 (socket says 172.21.15.2:60744) 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.757+0000 7f359a59c700 1 -- 172.21.15.2:0/1503351397 learned_addr learned my addr 172.21.15.2:0/1503351397 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.758+0000 7f359a59c700 1 -- 172.21.15.2:0/1503351397 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0094f0 con 0x7f359c076be0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.758+0000 7f359a59c700 1 --2- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c113a70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f358c0090b0 tx=0x7f358c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.758+0000 7f358affd700 1 -- 172.21.15.2:0/1503351397 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f358c0118f0 con 0x7f359c076be0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.758+0000 7f358affd700 1 -- 172.21.15.2:0/1503351397 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f358c010d10 con 0x7f359c076be0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.758+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c1141b0 con 0x7f359c076be0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.758+0000 7f358affd700 1 -- 172.21.15.2:0/1503351397 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f358c024e70 con 0x7f359c076be0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.758+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c114e50 con 0x7f359c076be0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.759+0000 7f358affd700 1 -- 172.21.15.2:0/1503351397 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f358c01a500 con 0x7f359c076be0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.759+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c05f5c0 con 0x7f359c076be0 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.759+0000 7f358affd700 1 --2- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3584036880 0x7f3584038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:08.663 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.759+0000 7f358affd700 1 -- 172.21.15.2:0/1503351397 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f358c04a6d0 con 0x7f359c076be0 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.759+0000 7f3599d9b700 1 --2- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3584036880 0x7f3584038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.759+0000 7f3599d9b700 1 --2- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3584036880 0x7f3584038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3590009990 tx=0x7f3590006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:07.763+0000 7f358affd700 1 -- 172.21.15.2:0/1503351397 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f358c010e80 con 0x7f359c076be0 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.062+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f359c04bed0 con 0x7f359c076be0 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.064+0000 7f358affd700 1 -- 172.21.15.2:0/1503351397 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f358c022340 con 0x7f359c076be0 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3584036880 msgr2=0x7f3584038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 --2- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3584036880 0x7f3584038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3590009990 tx=0x7f3590006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 msgr2=0x7f359c113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 --2- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c113a70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f358c0090b0 tx=0x7f358c009250 comp rx=0 tx=0).stop 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 shutdown_connections 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 --2- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3584036880 0x7f3584038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 --2- 172.21.15.2:0/1503351397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c076be0 0x7f359c113a70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 >> 172.21.15.2:0/1503351397 conn(0x7f359c0fcdd0 msgr2=0x7f359c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 shutdown_connections 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-02-16T22:13:08.066+0000 7f35a0ba0700 1 -- 172.21.15.2:0/1503351397 wait complete. 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout:Or, if you are only running a single cluster on this host: 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: ceph telemetry on 2024-02-16T22:13:08.666 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:08.666 INFO:teuthology.orchestra.run.smithi002.stdout:For more information see: 2024-02-16T22:13:08.666 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:08.666 INFO:teuthology.orchestra.run.smithi002.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-16T22:13:08.666 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:08.666 INFO:teuthology.orchestra.run.smithi002.stdout:Bootstrap complete. 2024-02-16T22:13:08.697 INFO:tasks.cephadm:Fetching config... 2024-02-16T22:13:08.697 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:13:08.697 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-16T22:13:08.754 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-16T22:13:08.754 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:13:08.754 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-16T22:13:08.809 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-16T22:13:08.810 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:13:08.810 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/keyring of=/dev/stdout 2024-02-16T22:13:08.878 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-16T22:13:08.878 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:13:08.878 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-16T22:13:08.933 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-16T22:13:08.933 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCp786tCKxNJcTonI75KKDqQK7CiWwyKhVH3wEdyNzEFsnqEFqaVA763z4Q45Pw/O46QLTnfgLOyfqKVdf71mAvUaOjLRfbjIXfw+WRaVczdCC6EIoKIfYSClNZ6tYVhSlLNiFNpc6VjcRR2BDA92Q8/Hr6YABMtWeQ7GMcmdOr2pEytZM/0vETbQsziwe4JUu7Xyqxu4WzpG2EmJewk1UFAwknxKwnpe90rqaG5kt85WJ7kkeJsGC3NgzEmVM5GECEh238lOqv57nAv909nrFwQECMIVNg0QaOE+tXkLNI3xrtSryuzrgo0UdW4Iapj7qpVxhoZw9lISeO0VlreUrpZMXNJ2OJROpWgVOqvYjvn99sZQRWqqVC1BEyuz6PgR6GfM3q4IaaiIK8IfuLvyAd831SFk0a/N9+K4bvClnZVR3AY3XhkdUIbCly9HPzcmEd0dGdZ0Ijx8jFywpahh3Sshmtv6e25CIxJuo5ZCYHDrnxz90td4PNcZXxsWjeDmE= ceph-43c11c36-cd18-11ee-95ba-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-16T22:13:09.013 INFO:teuthology.orchestra.run.smithi002.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCp786tCKxNJcTonI75KKDqQK7CiWwyKhVH3wEdyNzEFsnqEFqaVA763z4Q45Pw/O46QLTnfgLOyfqKVdf71mAvUaOjLRfbjIXfw+WRaVczdCC6EIoKIfYSClNZ6tYVhSlLNiFNpc6VjcRR2BDA92Q8/Hr6YABMtWeQ7GMcmdOr2pEytZM/0vETbQsziwe4JUu7Xyqxu4WzpG2EmJewk1UFAwknxKwnpe90rqaG5kt85WJ7kkeJsGC3NgzEmVM5GECEh238lOqv57nAv909nrFwQECMIVNg0QaOE+tXkLNI3xrtSryuzrgo0UdW4Iapj7qpVxhoZw9lISeO0VlreUrpZMXNJ2OJROpWgVOqvYjvn99sZQRWqqVC1BEyuz6PgR6GfM3q4IaaiIK8IfuLvyAd831SFk0a/N9+K4bvClnZVR3AY3XhkdUIbCly9HPzcmEd0dGdZ0Ijx8jFywpahh3Sshmtv6e25CIxJuo5ZCYHDrnxz90td4PNcZXxsWjeDmE= ceph-43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:13:09.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCp786tCKxNJcTonI75KKDqQK7CiWwyKhVH3wEdyNzEFsnqEFqaVA763z4Q45Pw/O46QLTnfgLOyfqKVdf71mAvUaOjLRfbjIXfw+WRaVczdCC6EIoKIfYSClNZ6tYVhSlLNiFNpc6VjcRR2BDA92Q8/Hr6YABMtWeQ7GMcmdOr2pEytZM/0vETbQsziwe4JUu7Xyqxu4WzpG2EmJewk1UFAwknxKwnpe90rqaG5kt85WJ7kkeJsGC3NgzEmVM5GECEh238lOqv57nAv909nrFwQECMIVNg0QaOE+tXkLNI3xrtSryuzrgo0UdW4Iapj7qpVxhoZw9lISeO0VlreUrpZMXNJ2OJROpWgVOqvYjvn99sZQRWqqVC1BEyuz6PgR6GfM3q4IaaiIK8IfuLvyAd831SFk0a/N9+K4bvClnZVR3AY3XhkdUIbCly9HPzcmEd0dGdZ0Ijx8jFywpahh3Sshmtv6e25CIxJuo5ZCYHDrnxz90td4PNcZXxsWjeDmE= ceph-43c11c36-cd18-11ee-95ba-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-16T22:13:09.072 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCp786tCKxNJcTonI75KKDqQK7CiWwyKhVH3wEdyNzEFsnqEFqaVA763z4Q45Pw/O46QLTnfgLOyfqKVdf71mAvUaOjLRfbjIXfw+WRaVczdCC6EIoKIfYSClNZ6tYVhSlLNiFNpc6VjcRR2BDA92Q8/Hr6YABMtWeQ7GMcmdOr2pEytZM/0vETbQsziwe4JUu7Xyqxu4WzpG2EmJewk1UFAwknxKwnpe90rqaG5kt85WJ7kkeJsGC3NgzEmVM5GECEh238lOqv57nAv909nrFwQECMIVNg0QaOE+tXkLNI3xrtSryuzrgo0UdW4Iapj7qpVxhoZw9lISeO0VlreUrpZMXNJ2OJROpWgVOqvYjvn99sZQRWqqVC1BEyuz6PgR6GfM3q4IaaiIK8IfuLvyAd831SFk0a/N9+K4bvClnZVR3AY3XhkdUIbCly9HPzcmEd0dGdZ0Ijx8jFywpahh3Sshmtv6e25CIxJuo5ZCYHDrnxz90td4PNcZXxsWjeDmE= ceph-43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:13:09.087 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-16T22:13:09.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:09 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1503351397' entity='client.admin' 2024-02-16T22:13:09.341 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:13:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.458+0000 7f7d0e497700 1 -- 172.21.15.2:0/4051567139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 msgr2=0x7f7d08070320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.458+0000 7f7d0e497700 1 --2- 172.21.15.2:0/4051567139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 0x7f7d08070320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80048c0 tx=0x7f7cf8009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.458+0000 7f7d0e497700 1 -- 172.21.15.2:0/4051567139 shutdown_connections 2024-02-16T22:13:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.458+0000 7f7d0e497700 1 --2- 172.21.15.2:0/4051567139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 0x7f7d08070320 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.458+0000 7f7d0e497700 1 -- 172.21.15.2:0/4051567139 >> 172.21.15.2:0/4051567139 conn(0x7f7d080f8270 msgr2=0x7f7d080fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:11.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.458+0000 7f7d0e497700 1 -- 172.21.15.2:0/4051567139 shutdown_connections 2024-02-16T22:13:11.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.458+0000 7f7d0e497700 1 -- 172.21.15.2:0/4051567139 wait complete. 2024-02-16T22:13:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.459+0000 7f7d0e497700 1 Processor -- start 2024-02-16T22:13:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.459+0000 7f7d0e497700 1 -- start start 2024-02-16T22:13:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.459+0000 7f7d0e497700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 0x7f7d0810f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.459+0000 7f7d0e497700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0810f5f0 con 0x7f7d08071eb0 2024-02-16T22:13:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d07fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 0x7f7d0810f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d07fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 0x7f7d0810f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45738/0 (socket says 172.21.15.2:45738) 2024-02-16T22:13:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d07fff700 1 -- 172.21.15.2:0/667181123 learned_addr learned my addr 172.21.15.2:0/667181123 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d07fff700 1 -- 172.21.15.2:0/667181123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf8005040 con 0x7f7d08071eb0 2024-02-16T22:13:11.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d07fff700 1 --2- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 0x7f7d0810f0b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009680 tx=0x7f7cf8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:11.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d04ff9700 1 -- 172.21.15.2:0/667181123 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf800b5c0 con 0x7f7d08071eb0 2024-02-16T22:13:11.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d04ff9700 1 -- 172.21.15.2:0/667181123 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cf8004e60 con 0x7f7d08071eb0 2024-02-16T22:13:11.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0810f7f0 con 0x7f7d08071eb0 2024-02-16T22:13:11.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d04ff9700 1 -- 172.21.15.2:0/667181123 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf801c590 con 0x7f7d08071eb0 2024-02-16T22:13:11.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.460+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d08110490 con 0x7f7d08071eb0 2024-02-16T22:13:11.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.461+0000 7f7d04ff9700 1 -- 172.21.15.2:0/667181123 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f7cf801c810 con 0x7f7d08071eb0 2024-02-16T22:13:11.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.462+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0805f5c0 con 0x7f7d08071eb0 2024-02-16T22:13:11.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.462+0000 7f7d04ff9700 1 --2- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7cf0036870 0x7f7cf0038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:11.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.462+0000 7f7d04ff9700 1 -- 172.21.15.2:0/667181123 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7cf8049b30 con 0x7f7d08071eb0 2024-02-16T22:13:11.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.462+0000 7f7d077fe700 1 --2- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7cf0036870 0x7f7cf0038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:11.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.462+0000 7f7d077fe700 1 --2- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7cf0036870 0x7f7cf0038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009990 tx=0x7f7cfc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:11.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.467+0000 7f7d04ff9700 1 -- 172.21.15.2:0/667181123 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7cf8023300 con 0x7f7d08071eb0 2024-02-16T22:13:11.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.664+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f7d0804bed0 con 0x7f7d08071eb0 2024-02-16T22:13:11.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.667+0000 7f7d04ff9700 1 -- 172.21.15.2:0/667181123 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7cf8011b60 con 0x7f7d08071eb0 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7cf0036870 msgr2=0x7f7cf0038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 --2- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7cf0036870 0x7f7cf0038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009990 tx=0x7f7cfc006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 msgr2=0x7f7d0810f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 --2- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 0x7f7d0810f0b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009680 tx=0x7f7cf8009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 shutdown_connections 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 --2- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f7cf0036870 0x7f7cf0038d20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 --2- 172.21.15.2:0/667181123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d08071eb0 0x7f7d0810f0b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 >> 172.21.15.2:0/667181123 conn(0x7f7d080f8270 msgr2=0x7f7d080f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 shutdown_connections 2024-02-16T22:13:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:11.669+0000 7f7d0e497700 1 -- 172.21.15.2:0/667181123 wait complete. 2024-02-16T22:13:12.225 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-16T22:13:12.225 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-16T22:13:12.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:12 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:12.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:12 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:12.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:12 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:13:12.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:12 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:12.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:12 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi002", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-16T22:13:12.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:12 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi002", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-16T22:13:12.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:12 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:13:12.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:12 smithi002 ceph-mon[29154]: Deploying daemon crash.smithi002 on smithi002 2024-02-16T22:13:12.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:12 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/667181123' entity='client.admin' 2024-02-16T22:13:12.514 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:13:15.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.498+0000 7f2d0aade700 1 -- 172.21.15.2:0/4169915606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 msgr2=0x7f2d040fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.498+0000 7f2d0aade700 1 --2- 172.21.15.2:0/4169915606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 0x7f2d040fbed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc009c60 tx=0x7f2cfc004d10 comp rx=0 tx=0).stop 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.498+0000 7f2d0aade700 1 -- 172.21.15.2:0/4169915606 shutdown_connections 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.498+0000 7f2d0aade700 1 --2- 172.21.15.2:0/4169915606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 0x7f2d040fbed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.498+0000 7f2d0aade700 1 -- 172.21.15.2:0/4169915606 >> 172.21.15.2:0/4169915606 conn(0x7f2d040f71a0 msgr2=0x7f2d040f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.498+0000 7f2d0aade700 1 -- 172.21.15.2:0/4169915606 shutdown_connections 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.498+0000 7f2d0aade700 1 -- 172.21.15.2:0/4169915606 wait complete. 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.499+0000 7f2d0aade700 1 Processor -- start 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.499+0000 7f2d0aade700 1 -- start start 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.499+0000 7f2d0aade700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 0x7f2d0410fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.499+0000 7f2d0aade700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040667f0 con 0x7f2d040fbac0 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.499+0000 7f2d0887a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 0x7f2d0410fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.499+0000 7f2d0887a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 0x7f2d0410fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45764/0 (socket says 172.21.15.2:45764) 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.499+0000 7f2d0887a700 1 -- 172.21.15.2:0/3717103647 learned_addr learned my addr 172.21.15.2:0/3717103647 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:15.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.499+0000 7f2d0887a700 1 -- 172.21.15.2:0/3717103647 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc009940 con 0x7f2d040fbac0 2024-02-16T22:13:15.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.500+0000 7f2d0887a700 1 --2- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 0x7f2d0410fc40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc015040 tx=0x7f2cfc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:15.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.500+0000 7f2d017fa700 1 -- 172.21.15.2:0/3717103647 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cfc007c80 con 0x7f2d040fbac0 2024-02-16T22:13:15.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.500+0000 7f2d017fa700 1 -- 172.21.15.2:0/3717103647 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cfc01d910 con 0x7f2d040fbac0 2024-02-16T22:13:15.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.500+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d040669f0 con 0x7f2d040fbac0 2024-02-16T22:13:15.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.500+0000 7f2d017fa700 1 -- 172.21.15.2:0/3717103647 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cfc0129f0 con 0x7f2d040fbac0 2024-02-16T22:13:15.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.500+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d041103c0 con 0x7f2d040fbac0 2024-02-16T22:13:15.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.501+0000 7f2d017fa700 1 -- 172.21.15.2:0/3717103647 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f2cfc019040 con 0x7f2d040fbac0 2024-02-16T22:13:15.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.501+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d04105890 con 0x7f2d040fbac0 2024-02-16T22:13:15.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.501+0000 7f2d017fa700 1 --2- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2cec036870 0x7f2cec038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:15.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.501+0000 7f2d017fa700 1 -- 172.21.15.2:0/3717103647 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2cfc04a3b0 con 0x7f2d040fbac0 2024-02-16T22:13:15.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.501+0000 7f2d03fff700 1 --2- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2cec036870 0x7f2cec038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:15.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.502+0000 7f2d03fff700 1 --2- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2cec036870 0x7f2cec038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8009990 tx=0x7f2cf8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:15.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.506+0000 7f2d017fa700 1 -- 172.21.15.2:0/3717103647 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2cfc012cb0 con 0x7f2d040fbac0 2024-02-16T22:13:15.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.710+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 --> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d0402b410 con 0x7f2cec036870 2024-02-16T22:13:15.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.713+0000 7f2d017fa700 1 -- 172.21.15.2:0/3717103647 <== mgr.14162 v2:172.21.15.2:6800/2481260931 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2d0402b410 con 0x7f2cec036870 2024-02-16T22:13:15.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.715+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2cec036870 msgr2=0x7f2cec038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:15.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.715+0000 7f2d0aade700 1 --2- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2cec036870 0x7f2cec038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8009990 tx=0x7f2cf8006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:15.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.715+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 msgr2=0x7f2d0410fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:15.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.715+0000 7f2d0aade700 1 --2- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 0x7f2d0410fc40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc015040 tx=0x7f2cfc004d10 comp rx=0 tx=0).stop 2024-02-16T22:13:15.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.716+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 shutdown_connections 2024-02-16T22:13:15.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.716+0000 7f2d0aade700 1 --2- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2cec036870 0x7f2cec038d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:15.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.716+0000 7f2d0aade700 1 --2- 172.21.15.2:0/3717103647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d040fbac0 0x7f2d0410fc40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:15.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.716+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 >> 172.21.15.2:0/3717103647 conn(0x7f2d040f71a0 msgr2=0x7f2d040f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:15.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.716+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 shutdown_connections 2024-02-16T22:13:15.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:15.716+0000 7f2d0aade700 1 -- 172.21.15.2:0/3717103647 wait complete. 2024-02-16T22:13:16.517 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi103 2024-02-16T22:13:16.517 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:13:16.518 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.conf 2024-02-16T22:13:16.536 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:13:16.536 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-16T22:13:16.594 INFO:tasks.cephadm:Adding host smithi103 to orchestrator... 2024-02-16T22:13:16.594 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch host add smithi103 2024-02-16T22:13:16.864 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:13:16.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:16 smithi002 ceph-mon[29154]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:13:16.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:16 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:18.107 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:17 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:18.108 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:17 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:18.108 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:17 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:18.108 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:17 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:18.108 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:17 smithi002 ceph-mon[29154]: Deploying daemon node-exporter.smithi002 on smithi002 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.361+0000 7f805d32b700 1 -- 172.21.15.2:0/3929662147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 msgr2=0x7f80580fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.361+0000 7f805d32b700 1 --2- 172.21.15.2:0/3929662147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 0x7f80580fcfe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f80480048c0 tx=0x7f8048009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.361+0000 7f805d32b700 1 -- 172.21.15.2:0/3929662147 shutdown_connections 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.361+0000 7f805d32b700 1 --2- 172.21.15.2:0/3929662147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 0x7f80580fcfe0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.361+0000 7f805d32b700 1 -- 172.21.15.2:0/3929662147 >> 172.21.15.2:0/3929662147 conn(0x7f80580f8270 msgr2=0x7f80580fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.361+0000 7f805d32b700 1 -- 172.21.15.2:0/3929662147 shutdown_connections 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.361+0000 7f805d32b700 1 -- 172.21.15.2:0/3929662147 wait complete. 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.362+0000 7f805d32b700 1 Processor -- start 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.362+0000 7f805d32b700 1 -- start start 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.362+0000 7f805d32b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 0x7f805810cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.362+0000 7f805d32b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805810d4a0 con 0x7f80580fcbd0 2024-02-16T22:13:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.362+0000 7f8056ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 0x7f805810cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:18.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.362+0000 7f8056ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 0x7f805810cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45788/0 (socket says 172.21.15.2:45788) 2024-02-16T22:13:18.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.362+0000 7f8056ffd700 1 -- 172.21.15.2:0/3461477082 learned_addr learned my addr 172.21.15.2:0/3461477082 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:18.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.363+0000 7f8056ffd700 1 -- 172.21.15.2:0/3461477082 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8048005040 con 0x7f80580fcbd0 2024-02-16T22:13:18.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.363+0000 7f8056ffd700 1 --2- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 0x7f805810cf60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8048009680 tx=0x7f8048009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:18.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.363+0000 7f803f7fe700 1 -- 172.21.15.2:0/3461477082 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f804800bed0 con 0x7f80580fcbd0 2024-02-16T22:13:18.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.363+0000 7f803f7fe700 1 -- 172.21.15.2:0/3461477082 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f804800b5c0 con 0x7f80580fcbd0 2024-02-16T22:13:18.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.363+0000 7f803f7fe700 1 -- 172.21.15.2:0/3461477082 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f804801c5b0 con 0x7f80580fcbd0 2024-02-16T22:13:18.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.363+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805810d6a0 con 0x7f80580fcbd0 2024-02-16T22:13:18.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.363+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805810dad0 con 0x7f80580fcbd0 2024-02-16T22:13:18.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.364+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805804bed0 con 0x7f80580fcbd0 2024-02-16T22:13:18.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.364+0000 7f803f7fe700 1 -- 172.21.15.2:0/3461477082 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f804801c7d0 con 0x7f80580fcbd0 2024-02-16T22:13:18.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.364+0000 7f803f7fe700 1 --2- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f8040036870 0x7f8040038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:18.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.364+0000 7f803f7fe700 1 -- 172.21.15.2:0/3461477082 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8048049c80 con 0x7f80580fcbd0 2024-02-16T22:13:18.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.364+0000 7f80567fc700 1 --2- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f8040036870 0x7f8040038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:18.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.365+0000 7f80567fc700 1 --2- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f8040036870 0x7f8040038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f804c009990 tx=0x7f804c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:18.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.369+0000 7f803f7fe700 1 -- 172.21.15.2:0/3461477082 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8048050080 con 0x7f80580fcbd0 2024-02-16T22:13:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:18.573+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 --> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}) v1 -- 0x7f80581068c0 con 0x7f8040036870 2024-02-16T22:13:19.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:19 smithi002 ceph-mon[29154]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:13:19.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:19 smithi002 ceph-mon[29154]: Deploying cephadm binary to smithi103 2024-02-16T22:13:20.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.712+0000 7f803f7fe700 1 -- 172.21.15.2:0/3461477082 <== mgr.14162 v2:172.21.15.2:6800/2481260931 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f80581068c0 con 0x7f8040036870 2024-02-16T22:13:20.714 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi103' with addr '172.21.15.103' 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f8040036870 msgr2=0x7f8040038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 --2- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f8040036870 0x7f8040038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f804c009990 tx=0x7f804c006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 msgr2=0x7f805810cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 --2- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 0x7f805810cf60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8048009680 tx=0x7f8048009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 shutdown_connections 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 --2- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f8040036870 0x7f8040038d20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 --2- 172.21.15.2:0/3461477082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80580fcbd0 0x7f805810cf60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 >> 172.21.15.2:0/3461477082 conn(0x7f80580f8270 msgr2=0x7f80580f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 shutdown_connections 2024-02-16T22:13:20.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:20.714+0000 7f805d32b700 1 -- 172.21.15.2:0/3461477082 wait complete. 2024-02-16T22:13:21.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:21 smithi002 ceph-mon[29154]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:21.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:21 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:21.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:21 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:21.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch host ls --format=json 2024-02-16T22:13:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:13:22.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:22 smithi002 ceph-mon[29154]: Added host smithi103 2024-02-16T22:13:23.445 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:23 smithi002 ceph-mon[29154]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:23.445 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:23 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:23.445 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:23 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:23.445 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:23 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:23.445 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:23 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:23.445 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:23 smithi002 ceph-mon[29154]: Deploying daemon alertmanager.smithi002 on smithi002 2024-02-16T22:13:24.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.084+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1526121965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 msgr2=0x7f3cdc0ffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.084+0000 7f3ce20d1700 1 --2- 172.21.15.2:0/1526121965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 0x7f3cdc0ffae0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0048c0 tx=0x7f3ccc009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.084+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1526121965 shutdown_connections 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.084+0000 7f3ce20d1700 1 --2- 172.21.15.2:0/1526121965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 0x7f3cdc0ffae0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.084+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1526121965 >> 172.21.15.2:0/1526121965 conn(0x7f3cdc0f71a0 msgr2=0x7f3cdc0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.084+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1526121965 shutdown_connections 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.084+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1526121965 wait complete. 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.085+0000 7f3ce20d1700 1 Processor -- start 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.085+0000 7f3ce20d1700 1 -- start start 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.085+0000 7f3ce20d1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 0x7f3cdc10cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.085+0000 7f3ce20d1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc10d3e0 con 0x7f3cdc0fd700 2024-02-16T22:13:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.085+0000 7f3cdb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 0x7f3cdc10cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.085+0000 7f3cdb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 0x7f3cdc10cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47618/0 (socket says 172.21.15.2:47618) 2024-02-16T22:13:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.085+0000 7f3cdb7fe700 1 -- 172.21.15.2:0/1109581192 learned_addr learned my addr 172.21.15.2:0/1109581192 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.086+0000 7f3cdb7fe700 1 -- 172.21.15.2:0/1109581192 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc005040 con 0x7f3cdc0fd700 2024-02-16T22:13:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.086+0000 7f3cdb7fe700 1 --2- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 0x7f3cdc10cea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009b80 tx=0x7f3ccc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.086+0000 7f3cd8ff9700 1 -- 172.21.15.2:0/1109581192 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc00b5c0 con 0x7f3cdc0fd700 2024-02-16T22:13:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.086+0000 7f3cd8ff9700 1 -- 172.21.15.2:0/1109581192 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ccc00be70 con 0x7f3cdc0fd700 2024-02-16T22:13:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.086+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cdc10d5e0 con 0x7f3cdc0fd700 2024-02-16T22:13:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.086+0000 7f3cd8ff9700 1 -- 172.21.15.2:0/1109581192 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc01c660 con 0x7f3cdc0fd700 2024-02-16T22:13:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.086+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc10e280 con 0x7f3cdc0fd700 2024-02-16T22:13:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.087+0000 7f3cd8ff9700 1 -- 172.21.15.2:0/1109581192 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f3ccc004d30 con 0x7f3cdc0fd700 2024-02-16T22:13:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.087+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cdc05f5c0 con 0x7f3cdc0fd700 2024-02-16T22:13:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.087+0000 7f3cd8ff9700 1 --2- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3cc8036870 0x7f3cc8038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.087+0000 7f3cd8ff9700 1 -- 172.21.15.2:0/1109581192 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3ccc049b30 con 0x7f3cdc0fd700 2024-02-16T22:13:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.087+0000 7f3cd3fff700 1 --2- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3cc8036870 0x7f3cc8038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.088+0000 7f3cd3fff700 1 --2- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3cc8036870 0x7f3cc8038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009990 tx=0x7f3cc4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:24.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.092+0000 7f3cd8ff9700 1 -- 172.21.15.2:0/1109581192 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3ccc011370 con 0x7f3cdc0fd700 2024-02-16T22:13:24.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.305+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 --> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3cdc10ee40 con 0x7f3cc8036870 2024-02-16T22:13:24.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.306+0000 7f3cd8ff9700 1 -- 172.21.15.2:0/1109581192 <== mgr.14162 v2:172.21.15.2:6800/2481260931 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f3cdc10ee40 con 0x7f3cc8036870 2024-02-16T22:13:24.309 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:13:24.309 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}] 2024-02-16T22:13:24.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.308+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3cc8036870 msgr2=0x7f3cc8038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:24.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.308+0000 7f3ce20d1700 1 --2- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3cc8036870 0x7f3cc8038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009990 tx=0x7f3cc4006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:24.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.308+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 msgr2=0x7f3cdc10cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:24.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.308+0000 7f3ce20d1700 1 --2- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 0x7f3cdc10cea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009b80 tx=0x7f3ccc009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:24.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.308+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 shutdown_connections 2024-02-16T22:13:24.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.308+0000 7f3ce20d1700 1 --2- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f3cc8036870 0x7f3cc8038d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:24.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.308+0000 7f3ce20d1700 1 --2- 172.21.15.2:0/1109581192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cdc0fd700 0x7f3cdc10cea0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:24.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.308+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 >> 172.21.15.2:0/1109581192 conn(0x7f3cdc0f71a0 msgr2=0x7f3cdc0f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:24.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.309+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 shutdown_connections 2024-02-16T22:13:24.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:24.309+0000 7f3ce20d1700 1 -- 172.21.15.2:0/1109581192 wait complete. 2024-02-16T22:13:25.223 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-16T22:13:25.223 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd crush tunables default 2024-02-16T22:13:25.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:25 smithi002 ceph-mon[29154]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:25.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:25 smithi002 ceph-mon[29154]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-16T22:13:25.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:25 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:25.499 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:13:27.631 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:27 smithi002 ceph-mon[29154]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.628+0000 7f063bad8700 1 -- 172.21.15.2:0/2259866748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 msgr2=0x7f06340ffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.628+0000 7f063bad8700 1 --2- 172.21.15.2:0/2259866748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 0x7f06340ffae0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f06300048c0 tx=0x7f0630009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.628+0000 7f063bad8700 1 -- 172.21.15.2:0/2259866748 shutdown_connections 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.628+0000 7f063bad8700 1 --2- 172.21.15.2:0/2259866748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 0x7f06340ffae0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.628+0000 7f063bad8700 1 -- 172.21.15.2:0/2259866748 >> 172.21.15.2:0/2259866748 conn(0x7f06340f71a0 msgr2=0x7f06340f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.629+0000 7f063bad8700 1 -- 172.21.15.2:0/2259866748 shutdown_connections 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.629+0000 7f063bad8700 1 -- 172.21.15.2:0/2259866748 wait complete. 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.629+0000 7f063bad8700 1 Processor -- start 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.629+0000 7f063bad8700 1 -- start start 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.629+0000 7f063bad8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 0x7f063410ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.629+0000 7f063bad8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063410d3f0 con 0x7f06340fd700 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f0639874700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 0x7f063410ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f0639874700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 0x7f063410ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47626/0 (socket says 172.21.15.2:47626) 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f0639874700 1 -- 172.21.15.2:0/839973598 learned_addr learned my addr 172.21.15.2:0/839973598 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f0639874700 1 -- 172.21.15.2:0/839973598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0630005040 con 0x7f06340fd700 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f0639874700 1 --2- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 0x7f063410ceb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0630009b80 tx=0x7f0630009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f062a7fc700 1 -- 172.21.15.2:0/839973598 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063000b5c0 con 0x7f06340fd700 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f062a7fc700 1 -- 172.21.15.2:0/839973598 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f063000be70 con 0x7f06340fd700 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063410d5f0 con 0x7f06340fd700 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f062a7fc700 1 -- 172.21.15.2:0/839973598 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f063001c660 con 0x7f06340fd700 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.630+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063410e290 con 0x7f06340fd700 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.631+0000 7f062a7fc700 1 -- 172.21.15.2:0/839973598 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f0630013420 con 0x7f06340fd700 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.631+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063405f5c0 con 0x7f06340fd700 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.631+0000 7f062a7fc700 1 --2- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f0620036870 0x7f0620038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.631+0000 7f062a7fc700 1 -- 172.21.15.2:0/839973598 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0630049a60 con 0x7f06340fd700 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.631+0000 7f0639073700 1 --2- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f0620036870 0x7f0620038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.632+0000 7f0639073700 1 --2- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f0620036870 0x7f0620038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0624009990 tx=0x7f0624006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:27.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.636+0000 7f062a7fc700 1 -- 172.21.15.2:0/839973598 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0630013b10 con 0x7f06340fd700 2024-02-16T22:13:27.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:27.844+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f063410ede0 con 0x7f06340fd700 2024-02-16T22:13:28.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.374+0000 7f062a7fc700 1 -- 172.21.15.2:0/839973598 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0630011370 con 0x7f06340fd700 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f0620036870 msgr2=0x7f0620038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 --2- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f0620036870 0x7f0620038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0624009990 tx=0x7f0624006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 msgr2=0x7f063410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 --2- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 0x7f063410ceb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0630009b80 tx=0x7f0630009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 shutdown_connections 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 --2- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f0620036870 0x7f0620038d20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 --2- 172.21.15.2:0/839973598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06340fd700 0x7f063410ceb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 >> 172.21.15.2:0/839973598 conn(0x7f06340f71a0 msgr2=0x7f06340f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 shutdown_connections 2024-02-16T22:13:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:13:28.376+0000 7f063bad8700 1 -- 172.21.15.2:0/839973598 wait complete. 2024-02-16T22:13:28.380 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2024-02-16T22:13:28.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:28 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/839973598' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-16T22:13:29.639 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:29 smithi002 ceph-mon[29154]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:29.639 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:29 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/839973598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-16T22:13:29.639 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:29 smithi002 ceph-mon[29154]: osdmap e4: 0 total, 0 up, 0 in 2024-02-16T22:13:29.640 INFO:tasks.cephadm:Adding mon.smithi002 on smithi002 2024-02-16T22:13:29.641 INFO:tasks.cephadm:Adding mon.smithi103 on smithi103 2024-02-16T22:13:29.641 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch apply mon '2;smithi002:172.21.15.2=smithi002;smithi103:172.21.15.103=smithi103' 2024-02-16T22:13:29.888 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:29.935 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:31.129 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:30 smithi002 ceph-mon[29154]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.430+0000 7f291721b700 1 -- 172.21.15.103:0/890947250 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 msgr2=0x7f29100fe610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.430+0000 7f291721b700 1 --2- 172.21.15.103:0/890947250 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 0x7f29100fe610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29040048c0 tx=0x7f2904009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.430+0000 7f291721b700 1 -- 172.21.15.103:0/890947250 shutdown_connections 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.430+0000 7f291721b700 1 --2- 172.21.15.103:0/890947250 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 0x7f29100fe610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.430+0000 7f291721b700 1 -- 172.21.15.103:0/890947250 >> 172.21.15.103:0/890947250 conn(0x7f29100f69c0 msgr2=0x7f29100f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.430+0000 7f291721b700 1 -- 172.21.15.103:0/890947250 shutdown_connections 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.430+0000 7f291721b700 1 -- 172.21.15.103:0/890947250 wait complete. 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.430+0000 7f291721b700 1 Processor -- start 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.430+0000 7f291721b700 1 -- start start 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.431+0000 7f291721b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 0x7f291010cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:31.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.431+0000 7f291721b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291010d480 con 0x7f29100fe200 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.431+0000 7f2914fb7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 0x7f291010cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.431+0000 7f2914fb7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 0x7f291010cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:53924/0 (socket says 172.21.15.103:53924) 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.431+0000 7f2914fb7700 1 -- 172.21.15.103:0/727944562 learned_addr learned my addr 172.21.15.103:0/727944562 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.431+0000 7f2914fb7700 1 -- 172.21.15.103:0/727944562 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2904005040 con 0x7f29100fe200 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.431+0000 7f2914fb7700 1 --2- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 0x7f291010cf40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2904009300 tx=0x7f2904009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.431+0000 7f290d7fa700 1 -- 172.21.15.103:0/727944562 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2904004e60 con 0x7f29100fe200 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.432+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291010d680 con 0x7f29100fe200 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.432+0000 7f290d7fa700 1 -- 172.21.15.103:0/727944562 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2904009680 con 0x7f29100fe200 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.432+0000 7f290d7fa700 1 -- 172.21.15.103:0/727944562 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f290401c5b0 con 0x7f29100fe200 2024-02-16T22:13:31.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.432+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291010e320 con 0x7f29100fe200 2024-02-16T22:13:31.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.432+0000 7f290d7fa700 1 -- 172.21.15.103:0/727944562 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f2904013420 con 0x7f29100fe200 2024-02-16T22:13:31.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.432+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291005f5c0 con 0x7f29100fe200 2024-02-16T22:13:31.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.433+0000 7f290d7fa700 1 --2- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2900036870 0x7f2900038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:31.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.433+0000 7f290d7fa700 1 -- 172.21.15.103:0/727944562 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2904049b20 con 0x7f29100fe200 2024-02-16T22:13:31.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.433+0000 7f290ffff700 1 --2- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2900036870 0x7f2900038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:31.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.433+0000 7f290ffff700 1 --2- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2900036870 0x7f2900038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f28fc009990 tx=0x7f28fc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:31.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.437+0000 7f290d7fa700 1 -- 172.21.15.103:0/727944562 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2904013980 con 0x7f29100fe200 2024-02-16T22:13:31.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.629+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 --> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi002:172.21.15.2=smithi002;smithi103:172.21.15.103=smithi103", "target": ["mon-mgr", ""]}) v1 -- 0x7f291010ef90 con 0x7f2900036870 2024-02-16T22:13:31.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.637+0000 7f290d7fa700 1 -- 172.21.15.103:0/727944562 <== mgr.14162 v2:172.21.15.2:6800/2481260931 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f291010ef90 con 0x7f2900036870 2024-02-16T22:13:31.638 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled mon update... 2024-02-16T22:13:31.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2900036870 msgr2=0x7f2900038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:31.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 --2- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2900036870 0x7f2900038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f28fc009990 tx=0x7f28fc006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:31.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 msgr2=0x7f291010cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:31.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 --2- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 0x7f291010cf40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2904009300 tx=0x7f2904009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:31.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 shutdown_connections 2024-02-16T22:13:31.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 --2- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f2900036870 0x7f2900038d20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:31.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 --2- 172.21.15.103:0/727944562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29100fe200 0x7f291010cf40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:31.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 >> 172.21.15.103:0/727944562 conn(0x7f29100f69c0 msgr2=0x7f29100f7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:31.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 shutdown_connections 2024-02-16T22:13:31.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:31.638+0000 7f291721b700 1 -- 172.21.15.103:0/727944562 wait complete. 2024-02-16T22:13:32.286 DEBUG:teuthology.orchestra.run.smithi103:mon.smithi103> sudo journalctl -f -n 0 -u ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mon.smithi103.service 2024-02-16T22:13:32.289 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:13:32.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:13:32.566 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:32.612 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:32.795 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:32 smithi002 ceph-mon[29154]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi002:172.21.15.2=smithi002;smithi103:172.21.15.103=smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:13:32.795 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:32 smithi002 ceph-mon[29154]: Saving service mon spec with placement smithi002:172.21.15.2=smithi002;smithi103:172.21.15.103=smithi103;count:2 2024-02-16T22:13:32.795 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:32 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:32.795 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:32 smithi002 ceph-mon[29154]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:34.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.041+0000 7f7498bf5700 1 -- 172.21.15.103:0/3947245371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 msgr2=0x7f74940fe610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.041+0000 7f7498bf5700 1 --2- 172.21.15.103:0/3947245371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 0x7f74940fe610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f74880048c0 tx=0x7f7488009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.041+0000 7f7498bf5700 1 -- 172.21.15.103:0/3947245371 shutdown_connections 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.041+0000 7f7498bf5700 1 --2- 172.21.15.103:0/3947245371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 0x7f74940fe610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.041+0000 7f7498bf5700 1 -- 172.21.15.103:0/3947245371 >> 172.21.15.103:0/3947245371 conn(0x7f74940f69c0 msgr2=0x7f74940f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.041+0000 7f7498bf5700 1 -- 172.21.15.103:0/3947245371 shutdown_connections 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.041+0000 7f7498bf5700 1 -- 172.21.15.103:0/3947245371 wait complete. 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.042+0000 7f7498bf5700 1 Processor -- start 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.042+0000 7f7498bf5700 1 -- start start 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.042+0000 7f7498bf5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 0x7f74940fda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.042+0000 7f7498bf5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74940fbec0 con 0x7f74940fe200 2024-02-16T22:13:34.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.042+0000 7f749259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 0x7f74940fda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.042+0000 7f749259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 0x7f74940fda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:53942/0 (socket says 172.21.15.103:53942) 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.042+0000 7f749259c700 1 -- 172.21.15.103:0/6933436 learned_addr learned my addr 172.21.15.103:0/6933436 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.043+0000 7f749259c700 1 -- 172.21.15.103:0/6933436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7488005040 con 0x7f74940fe200 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.043+0000 7f749259c700 1 --2- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 0x7f74940fda40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7488009330 tx=0x7f7488009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.043+0000 7f7482ffd700 1 -- 172.21.15.103:0/6933436 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f748800be50 con 0x7f74940fe200 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.043+0000 7f7482ffd700 1 -- 172.21.15.103:0/6933436 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f74880096b0 con 0x7f74940fe200 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.043+0000 7f7482ffd700 1 -- 172.21.15.103:0/6933436 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f748801c5b0 con 0x7f74940fe200 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.043+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74940fc0c0 con 0x7f74940fe200 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.043+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74940fc5b0 con 0x7f74940fe200 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.044+0000 7f7482ffd700 1 -- 172.21.15.103:0/6933436 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f7488013420 con 0x7f74940fe200 2024-02-16T22:13:34.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.044+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7494106910 con 0x7f74940fe200 2024-02-16T22:13:34.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.044+0000 7f7482ffd700 1 --2- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f747c036870 0x7f747c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:34.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.044+0000 7f7482ffd700 1 -- 172.21.15.103:0/6933436 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7488049a40 con 0x7f74940fe200 2024-02-16T22:13:34.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.044+0000 7f7491d9b700 1 --2- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f747c036870 0x7f747c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:34.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.045+0000 7f7491d9b700 1 --2- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f747c036870 0x7f747c038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7484009990 tx=0x7f7484006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:34.052 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.049+0000 7f7482ffd700 1 -- 172.21.15.103:0/6933436 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f74880139f0 con 0x7f74940fe200 2024-02-16T22:13:34.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.306+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f749404bed0 con 0x7f74940fe200 2024-02-16T22:13:34.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.306+0000 7f7482ffd700 1 -- 172.21.15.103:0/6933436 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f7488011370 con 0x7f74940fe200 2024-02-16T22:13:34.309 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:13:34.310 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:13:34.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f747c036870 msgr2=0x7f747c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:34.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 --2- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f747c036870 0x7f747c038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7484009990 tx=0x7f7484006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:34.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 msgr2=0x7f74940fda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:34.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 --2- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 0x7f74940fda40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7488009330 tx=0x7f7488009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:34.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 shutdown_connections 2024-02-16T22:13:34.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 --2- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f747c036870 0x7f747c038d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:34.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 --2- 172.21.15.103:0/6933436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74940fe200 0x7f74940fda40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:34.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 >> 172.21.15.103:0/6933436 conn(0x7f74940f69c0 msgr2=0x7f74940f7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:34.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 shutdown_connections 2024-02-16T22:13:34.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:34.308+0000 7f7498bf5700 1 -- 172.21.15.103:0/6933436 wait complete. 2024-02-16T22:13:34.313 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:13:34.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:34.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:34.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:34.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:34.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:34.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:34.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-16T22:13:34.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-16T22:13:34.477 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:34.477 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:34 smithi002 ceph-mon[29154]: Deploying daemon grafana.smithi002 on smithi002 2024-02-16T22:13:35.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:35 smithi002 ceph-mon[29154]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:35.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:35 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/6933436' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:13:35.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:35 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:35.864 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:13:35.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:13:36.114 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:36.162 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:37.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:37 smithi002 ceph-mon[29154]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.482+0000 7ff51da68700 1 -- 172.21.15.103:0/146025575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 msgr2=0x7ff5180fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.482+0000 7ff51da68700 1 --2- 172.21.15.103:0/146025575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 0x7ff5180fcfe0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0048c0 tx=0x7ff50c009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.482+0000 7ff51da68700 1 -- 172.21.15.103:0/146025575 shutdown_connections 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.482+0000 7ff51da68700 1 --2- 172.21.15.103:0/146025575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 0x7ff5180fcfe0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.482+0000 7ff51da68700 1 -- 172.21.15.103:0/146025575 >> 172.21.15.103:0/146025575 conn(0x7ff5180f8290 msgr2=0x7ff5180fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.482+0000 7ff51da68700 1 -- 172.21.15.103:0/146025575 shutdown_connections 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.482+0000 7ff51da68700 1 -- 172.21.15.103:0/146025575 wait complete. 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.483+0000 7ff51da68700 1 Processor -- start 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.483+0000 7ff51da68700 1 -- start start 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.483+0000 7ff51da68700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 0x7ff518071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.483+0000 7ff51da68700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180702e0 con 0x7ff5180fcbd0 2024-02-16T22:13:37.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.483+0000 7ff5177fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 0x7ff518071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:37.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.483+0000 7ff5177fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 0x7ff518071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:45544/0 (socket says 172.21.15.103:45544) 2024-02-16T22:13:37.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.483+0000 7ff5177fe700 1 -- 172.21.15.103:0/3964559962 learned_addr learned my addr 172.21.15.103:0/3964559962 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:13:37.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.484+0000 7ff5177fe700 1 -- 172.21.15.103:0/3964559962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c005040 con 0x7ff5180fcbd0 2024-02-16T22:13:37.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.484+0000 7ff5177fe700 1 --2- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 0x7ff518071e60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009b80 tx=0x7ff50c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:37.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.484+0000 7ff4fffff700 1 -- 172.21.15.103:0/3964559962 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c00b5c0 con 0x7ff5180fcbd0 2024-02-16T22:13:37.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.484+0000 7ff4fffff700 1 -- 172.21.15.103:0/3964559962 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff50c00be70 con 0x7ff5180fcbd0 2024-02-16T22:13:37.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.484+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5180704e0 con 0x7ff5180fcbd0 2024-02-16T22:13:37.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.484+0000 7ff4fffff700 1 -- 172.21.15.103:0/3964559962 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50c01c660 con 0x7ff5180fcbd0 2024-02-16T22:13:37.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.484+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5180709d0 con 0x7ff5180fcbd0 2024-02-16T22:13:37.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.485+0000 7ff4fffff700 1 -- 172.21.15.103:0/3964559962 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7ff50c01c8d0 con 0x7ff5180fcbd0 2024-02-16T22:13:37.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.485+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5181069a0 con 0x7ff5180fcbd0 2024-02-16T22:13:37.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.485+0000 7ff4fffff700 1 --2- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7ff500036870 0x7ff500038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:37.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.485+0000 7ff4fffff700 1 -- 172.21.15.103:0/3964559962 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff50c049bb0 con 0x7ff5180fcbd0 2024-02-16T22:13:37.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.485+0000 7ff516ffd700 1 --2- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7ff500036870 0x7ff500038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:37.489 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.486+0000 7ff516ffd700 1 --2- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7ff500036870 0x7ff500038d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff508009990 tx=0x7ff508006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:37.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.490+0000 7ff4fffff700 1 -- 172.21.15.103:0/3964559962 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff50c013420 con 0x7ff5180fcbd0 2024-02-16T22:13:37.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.743+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff51804bed0 con 0x7ff5180fcbd0 2024-02-16T22:13:37.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.743+0000 7ff4fffff700 1 -- 172.21.15.103:0/3964559962 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7ff50c011370 con 0x7ff5180fcbd0 2024-02-16T22:13:37.746 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:13:37.747 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:13:37.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.745+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7ff500036870 msgr2=0x7ff500038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:37.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.745+0000 7ff51da68700 1 --2- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7ff500036870 0x7ff500038d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff508009990 tx=0x7ff508006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:37.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.745+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 msgr2=0x7ff518071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.745+0000 7ff51da68700 1 --2- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 0x7ff518071e60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009b80 tx=0x7ff50c009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.745+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 shutdown_connections 2024-02-16T22:13:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.745+0000 7ff51da68700 1 --2- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7ff500036870 0x7ff500038d20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.745+0000 7ff51da68700 1 --2- 172.21.15.103:0/3964559962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5180fcbd0 0x7ff518071e60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.745+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 >> 172.21.15.103:0/3964559962 conn(0x7ff5180f8290 msgr2=0x7ff5180f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.746+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 shutdown_connections 2024-02-16T22:13:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:37.746+0000 7ff51da68700 1 -- 172.21.15.103:0/3964559962 wait complete. 2024-02-16T22:13:37.749 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:13:38.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:38 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/3964559962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:13:39.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:13:39.368 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:13:39.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:39 smithi002 ceph-mon[29154]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:39.617 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:39.664 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.145+0000 7fd39d0f5700 1 -- 172.21.15.103:0/342499041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 msgr2=0x7fd398070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.145+0000 7fd39d0f5700 1 --2- 172.21.15.103:0/342499041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 0x7fd398070380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd390009c60 tx=0x7fd390004d10 comp rx=0 tx=0).stop 2024-02-16T22:13:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.145+0000 7fd39d0f5700 1 -- 172.21.15.103:0/342499041 shutdown_connections 2024-02-16T22:13:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.145+0000 7fd39d0f5700 1 --2- 172.21.15.103:0/342499041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 0x7fd398070380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.145+0000 7fd39d0f5700 1 -- 172.21.15.103:0/342499041 >> 172.21.15.103:0/342499041 conn(0x7fd3980f82a0 msgr2=0x7fd3980fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.145+0000 7fd39d0f5700 1 -- 172.21.15.103:0/342499041 shutdown_connections 2024-02-16T22:13:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.145+0000 7fd39d0f5700 1 -- 172.21.15.103:0/342499041 wait complete. 2024-02-16T22:13:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.146+0000 7fd39d0f5700 1 Processor -- start 2024-02-16T22:13:41.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.146+0000 7fd39d0f5700 1 -- start start 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.146+0000 7fd39d0f5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 0x7fd39810f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.146+0000 7fd39d0f5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39810f5c0 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.146+0000 7fd396d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 0x7fd39810f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.146+0000 7fd396d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 0x7fd39810f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:45562/0 (socket says 172.21.15.103:45562) 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.146+0000 7fd396d9d700 1 -- 172.21.15.103:0/4207948971 learned_addr learned my addr 172.21.15.103:0/4207948971 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.146+0000 7fd396d9d700 1 -- 172.21.15.103:0/4207948971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd390009940 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.147+0000 7fd396d9d700 1 --2- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 0x7fd39810f080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd390004e60 tx=0x7fd390004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.147+0000 7fd37f7fe700 1 -- 172.21.15.103:0/4207948971 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd39001d440 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.147+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39810f7c0 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.147+0000 7fd37f7fe700 1 -- 172.21.15.103:0/4207948971 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd39001da70 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.147+0000 7fd37f7fe700 1 -- 172.21.15.103:0/4207948971 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd390012a30 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.147+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd398110460 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.148+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39805f5c0 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.148+0000 7fd37f7fe700 1 -- 172.21.15.103:0/4207948971 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fd390019040 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.148+0000 7fd37f7fe700 1 --2- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fd380036870 0x7fd380038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.148+0000 7fd37f7fe700 1 -- 172.21.15.103:0/4207948971 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd39004a3f0 con 0x7fd398071f10 2024-02-16T22:13:41.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.148+0000 7fd39659c700 1 --2- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fd380036870 0x7fd380038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:41.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.149+0000 7fd39659c700 1 --2- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fd380036870 0x7fd380038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009990 tx=0x7fd38c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:41.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.153+0000 7fd37f7fe700 1 -- 172.21.15.103:0/4207948971 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd39002eb40 con 0x7fd398071f10 2024-02-16T22:13:41.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:40 smithi002 ceph-mon[29154]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:41.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.408+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd398110f60 con 0x7fd398071f10 2024-02-16T22:13:41.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.408+0000 7fd37f7fe700 1 -- 172.21.15.103:0/4207948971 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fd390017070 con 0x7fd398071f10 2024-02-16T22:13:41.411 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:13:41.412 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:13:41.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fd380036870 msgr2=0x7fd380038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:41.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 --2- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fd380036870 0x7fd380038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009990 tx=0x7fd38c006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:41.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 msgr2=0x7fd39810f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:41.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 --2- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 0x7fd39810f080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd390004e60 tx=0x7fd390004d10 comp rx=0 tx=0).stop 2024-02-16T22:13:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 shutdown_connections 2024-02-16T22:13:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 --2- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fd380036870 0x7fd380038d20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 --2- 172.21.15.103:0/4207948971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd398071f10 0x7fd39810f080 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 >> 172.21.15.103:0/4207948971 conn(0x7fd3980f82a0 msgr2=0x7fd3980f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 shutdown_connections 2024-02-16T22:13:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:41.410+0000 7fd39d0f5700 1 -- 172.21.15.103:0/4207948971 wait complete. 2024-02-16T22:13:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:13:42.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:41 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/4207948971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:13:42.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:13:42.946 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:13:43.092 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:42 smithi002 ceph-mon[29154]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:43.195 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:43.241 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:44.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.894+0000 7f5b116a0700 1 -- 172.21.15.103:0/2103870802 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 msgr2=0x7f5b0c0fb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.894+0000 7f5b116a0700 1 --2- 172.21.15.103:0/2103870802 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 0x7f5b0c0fb290 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0048c0 tx=0x7f5afc009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.894+0000 7f5b116a0700 1 -- 172.21.15.103:0/2103870802 shutdown_connections 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.894+0000 7f5b116a0700 1 --2- 172.21.15.103:0/2103870802 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 0x7f5b0c0fb290 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.894+0000 7f5b116a0700 1 -- 172.21.15.103:0/2103870802 >> 172.21.15.103:0/2103870802 conn(0x7f5b0c0f6920 msgr2=0x7f5b0c0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.894+0000 7f5b116a0700 1 -- 172.21.15.103:0/2103870802 shutdown_connections 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.894+0000 7f5b116a0700 1 -- 172.21.15.103:0/2103870802 wait complete. 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.895+0000 7f5b116a0700 1 Processor -- start 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.895+0000 7f5b116a0700 1 -- start start 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.895+0000 7f5b116a0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 0x7f5b0c10cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:44.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.895+0000 7f5b116a0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c10d420 con 0x7f5b0c0fae80 2024-02-16T22:13:44.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.895+0000 7f5b0affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 0x7f5b0c10cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:44.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.895+0000 7f5b0affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 0x7f5b0c10cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:45588/0 (socket says 172.21.15.103:45588) 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.895+0000 7f5b0affd700 1 -- 172.21.15.103:0/4011184044 learned_addr learned my addr 172.21.15.103:0/4011184044 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.896+0000 7f5b0affd700 1 -- 172.21.15.103:0/4011184044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc005040 con 0x7f5b0c0fae80 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.896+0000 7f5b0affd700 1 --2- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 0x7f5b0c10cee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009040 tx=0x7f5afc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.896+0000 7f5af37fe700 1 -- 172.21.15.103:0/4011184044 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5afc00bed0 con 0x7f5b0c0fae80 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.896+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0c10d620 con 0x7f5b0c0fae80 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.896+0000 7f5af37fe700 1 -- 172.21.15.103:0/4011184044 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5afc00b5c0 con 0x7f5b0c0fae80 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.896+0000 7f5af37fe700 1 -- 172.21.15.103:0/4011184044 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5afc01c5b0 con 0x7f5b0c0fae80 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.896+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0c10e2c0 con 0x7f5b0c0fae80 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.897+0000 7f5af37fe700 1 -- 172.21.15.103:0/4011184044 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f5afc013420 con 0x7f5b0c0fae80 2024-02-16T22:13:44.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.897+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0c05f5c0 con 0x7f5b0c0fae80 2024-02-16T22:13:44.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.897+0000 7f5af37fe700 1 --2- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5af4036870 0x7f5af4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:44.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.897+0000 7f5af37fe700 1 -- 172.21.15.103:0/4011184044 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5afc049b70 con 0x7f5b0c0fae80 2024-02-16T22:13:44.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.898+0000 7f5b0a7fc700 1 --2- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5af4036870 0x7f5af4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:44.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.898+0000 7f5b0a7fc700 1 --2- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5af4036870 0x7f5af4038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009990 tx=0x7f5b00006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:44.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:44.903+0000 7f5af37fe700 1 -- 172.21.15.103:0/4011184044 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5afc013a80 con 0x7f5b0c0fae80 2024-02-16T22:13:45.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.157+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b0c10ee10 con 0x7f5b0c0fae80 2024-02-16T22:13:45.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.158+0000 7f5af37fe700 1 -- 172.21.15.103:0/4011184044 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f5afc011370 con 0x7f5b0c0fae80 2024-02-16T22:13:45.160 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:13:45.160 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5af4036870 msgr2=0x7f5af4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 --2- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5af4036870 0x7f5af4038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009990 tx=0x7f5b00006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 msgr2=0x7f5b0c10cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 --2- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 0x7f5b0c10cee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009040 tx=0x7f5afc009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 shutdown_connections 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 --2- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5af4036870 0x7f5af4038d20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 --2- 172.21.15.103:0/4011184044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b0c0fae80 0x7f5b0c10cee0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 >> 172.21.15.103:0/4011184044 conn(0x7f5b0c0f6920 msgr2=0x7f5b0c0f76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 shutdown_connections 2024-02-16T22:13:45.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:45.160+0000 7f5b116a0700 1 -- 172.21.15.103:0/4011184044 wait complete. 2024-02-16T22:13:45.163 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:13:45.280 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:45 smithi002 ceph-mon[29154]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:45.280 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:45 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/4011184044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:13:46.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:13:46.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:13:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:47.055 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:47.205 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:47 smithi002 ceph-mon[29154]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:47.206 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:47 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:47.206 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:47 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:47.206 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:47 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:47.206 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:47 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:47.206 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:47 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:47.206 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:47 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:48.297 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:48 smithi002 ceph-mon[29154]: Deploying daemon prometheus.smithi002 on smithi002 2024-02-16T22:13:48.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.472+0000 7f76eb262700 1 -- 172.21.15.103:0/2258211098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 msgr2=0x7f76e40ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:48.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.472+0000 7f76eb262700 1 --2- 172.21.15.103:0/2258211098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 0x7f76e40ff790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0048c0 tx=0x7f76cc009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:48.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.472+0000 7f76eb262700 1 -- 172.21.15.103:0/2258211098 shutdown_connections 2024-02-16T22:13:48.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.472+0000 7f76eb262700 1 --2- 172.21.15.103:0/2258211098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 0x7f76e40ff790 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:48.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.472+0000 7f76eb262700 1 -- 172.21.15.103:0/2258211098 >> 172.21.15.103:0/2258211098 conn(0x7f76e40f71a0 msgr2=0x7f76e40f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:48.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.472+0000 7f76eb262700 1 -- 172.21.15.103:0/2258211098 shutdown_connections 2024-02-16T22:13:48.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.472+0000 7f76eb262700 1 -- 172.21.15.103:0/2258211098 wait complete. 2024-02-16T22:13:48.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.473+0000 7f76eb262700 1 Processor -- start 2024-02-16T22:13:48.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.473+0000 7f76eb262700 1 -- start start 2024-02-16T22:13:48.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.473+0000 7f76eb262700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 0x7f76e40fd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:48.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.473+0000 7f76eb262700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e40fb690 con 0x7f76e4066660 2024-02-16T22:13:48.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.473+0000 7f76e8ffe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 0x7f76e40fd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:48.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.473+0000 7f76e8ffe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 0x7f76e40fd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:35158/0 (socket says 172.21.15.103:35158) 2024-02-16T22:13:48.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.473+0000 7f76e8ffe700 1 -- 172.21.15.103:0/4293791560 learned_addr learned my addr 172.21.15.103:0/4293791560 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:13:48.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.474+0000 7f76e8ffe700 1 -- 172.21.15.103:0/4293791560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc005040 con 0x7f76e4066660 2024-02-16T22:13:48.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.474+0000 7f76e8ffe700 1 --2- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 0x7f76e40fd210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009b80 tx=0x7f76cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:48.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.474+0000 7f76e17fa700 1 -- 172.21.15.103:0/4293791560 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76cc00b5c0 con 0x7f76e4066660 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.474+0000 7f76e17fa700 1 -- 172.21.15.103:0/4293791560 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76cc00be70 con 0x7f76e4066660 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.474+0000 7f76e17fa700 1 -- 172.21.15.103:0/4293791560 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76cc01c660 con 0x7f76e4066660 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.474+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e40fb890 con 0x7f76e4066660 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.474+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e40fbd80 con 0x7f76e4066660 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.475+0000 7f76e17fa700 1 -- 172.21.15.103:0/4293791560 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f76cc013420 con 0x7f76e4066660 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.475+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e4106980 con 0x7f76e4066660 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.475+0000 7f76e17fa700 1 --2- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f76d00368c0 0x7f76d0038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.475+0000 7f76e17fa700 1 -- 172.21.15.103:0/4293791560 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f76cc0499f0 con 0x7f76e4066660 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.475+0000 7f76e3fff700 1 --2- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f76d00368c0 0x7f76d0038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:48.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.476+0000 7f76e3fff700 1 --2- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f76d00368c0 0x7f76d0038d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009990 tx=0x7f76d4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:48.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.484+0000 7f76e17fa700 1 -- 172.21.15.103:0/4293791560 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f76cc013b10 con 0x7f76e4066660 2024-02-16T22:13:48.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.737+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76e404bed0 con 0x7f76e4066660 2024-02-16T22:13:48.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.738+0000 7f76e17fa700 1 -- 172.21.15.103:0/4293791560 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f76cc011370 con 0x7f76e4066660 2024-02-16T22:13:48.740 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:13:48.740 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f76d00368c0 msgr2=0x7f76d0038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 --2- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f76d00368c0 0x7f76d0038d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009990 tx=0x7f76d4006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 msgr2=0x7f76e40fd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 --2- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 0x7f76e40fd210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009b80 tx=0x7f76cc009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 shutdown_connections 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 --2- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f76d00368c0 0x7f76d0038d70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 --2- 172.21.15.103:0/4293791560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76e4066660 0x7f76e40fd210 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 >> 172.21.15.103:0/4293791560 conn(0x7f76e40f71a0 msgr2=0x7f76e40f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 shutdown_connections 2024-02-16T22:13:48.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:48.740+0000 7f76eb262700 1 -- 172.21.15.103:0/4293791560 wait complete. 2024-02-16T22:13:48.743 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:13:49.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:49 smithi002 ceph-mon[29154]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:49.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:49 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/4293791560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:13:50.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:13:50.278 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:13:50.543 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:50.591 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:51.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:51 smithi002 ceph-mon[29154]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:51.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:51 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:13:51.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.981+0000 7f5cc3608700 1 -- 172.21.15.103:0/2414620387 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 msgr2=0x7f5cbc0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:51.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.981+0000 7f5cc3608700 1 --2- 172.21.15.103:0/2414620387 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 0x7f5cbc0fcfe0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80048c0 tx=0x7f5cb8009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:51.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.981+0000 7f5cc3608700 1 -- 172.21.15.103:0/2414620387 shutdown_connections 2024-02-16T22:13:51.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.981+0000 7f5cc3608700 1 --2- 172.21.15.103:0/2414620387 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 0x7f5cbc0fcfe0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:51.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.981+0000 7f5cc3608700 1 -- 172.21.15.103:0/2414620387 >> 172.21.15.103:0/2414620387 conn(0x7f5cbc0f8290 msgr2=0x7f5cbc0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:51.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.981+0000 7f5cc3608700 1 -- 172.21.15.103:0/2414620387 shutdown_connections 2024-02-16T22:13:51.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.981+0000 7f5cc3608700 1 -- 172.21.15.103:0/2414620387 wait complete. 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.982+0000 7f5cc3608700 1 Processor -- start 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.982+0000 7f5cc3608700 1 -- start start 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.982+0000 7f5cc3608700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 0x7f5cbc10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.982+0000 7f5cc3608700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc10d400 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.982+0000 7f5cc13a4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 0x7f5cbc10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.982+0000 7f5cc13a4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 0x7f5cbc10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:35174/0 (socket says 172.21.15.103:35174) 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.982+0000 7f5cc13a4700 1 -- 172.21.15.103:0/259101024 learned_addr learned my addr 172.21.15.103:0/259101024 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.982+0000 7f5cc13a4700 1 -- 172.21.15.103:0/259101024 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8005040 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.982+0000 7f5cc13a4700 1 --2- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 0x7f5cbc10cec0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8009680 tx=0x7f5cb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.983+0000 7f5cb1ffb700 1 -- 172.21.15.103:0/259101024 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb800bed0 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.983+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cbc10d600 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.983+0000 7f5cb1ffb700 1 -- 172.21.15.103:0/259101024 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5cb800b5c0 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.983+0000 7f5cb1ffb700 1 -- 172.21.15.103:0/259101024 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cb801c5b0 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.983+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cbc10e2a0 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.984+0000 7f5cb1ffb700 1 -- 172.21.15.103:0/259101024 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f5cb8013420 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.984+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cbc05f5c0 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.984+0000 7f5cb1ffb700 1 --2- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5ca8036820 0x7f5ca8038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:51.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.984+0000 7f5cb1ffb700 1 -- 172.21.15.103:0/259101024 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5cb8049ad0 con 0x7f5cbc0fcbd0 2024-02-16T22:13:51.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.984+0000 7f5cc0ba3700 1 --2- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5ca8036820 0x7f5ca8038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:51.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.984+0000 7f5cc0ba3700 1 --2- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5ca8036820 0x7f5ca8038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5cac009990 tx=0x7f5cac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:51.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:51.990+0000 7f5cb1ffb700 1 -- 172.21.15.103:0/259101024 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5cb8025400 con 0x7f5cbc0fcbd0 2024-02-16T22:13:52.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.247+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cbc05f5c0 con 0x7f5cbc0fcbd0 2024-02-16T22:13:52.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.248+0000 7f5cb1ffb700 1 -- 172.21.15.103:0/259101024 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f5cb8004d30 con 0x7f5cbc0fcbd0 2024-02-16T22:13:52.250 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:13:52.250 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:13:52.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5ca8036820 msgr2=0x7f5ca8038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:52.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 --2- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5ca8036820 0x7f5ca8038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5cac009990 tx=0x7f5cac006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:52.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 msgr2=0x7f5cbc10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:52.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 --2- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 0x7f5cbc10cec0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8009680 tx=0x7f5cb8009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:52.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 shutdown_connections 2024-02-16T22:13:52.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 --2- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5ca8036820 0x7f5ca8038cd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:52.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 --2- 172.21.15.103:0/259101024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc0fcbd0 0x7f5cbc10cec0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:52.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 >> 172.21.15.103:0/259101024 conn(0x7f5cbc0f8290 msgr2=0x7f5cbc0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:52.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 shutdown_connections 2024-02-16T22:13:52.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:52.250+0000 7f5cc3608700 1 -- 172.21.15.103:0/259101024 wait complete. 2024-02-16T22:13:52.253 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:13:53.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:53 smithi002 ceph-mon[29154]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:53.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:53 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/259101024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:13:53.824 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:13:53.824 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:13:54.072 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:54.119 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:55.380 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:55 smithi002 ceph-mon[29154]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:55.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.410+0000 7fa825a88700 1 -- 172.21.15.103:0/90573792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 msgr2=0x7fa8200ffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.410+0000 7fa825a88700 1 --2- 172.21.15.103:0/90573792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 0x7fa8200ffae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa8100048c0 tx=0x7fa810009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.410+0000 7fa825a88700 1 -- 172.21.15.103:0/90573792 shutdown_connections 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.410+0000 7fa825a88700 1 --2- 172.21.15.103:0/90573792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 0x7fa8200ffae0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.410+0000 7fa825a88700 1 -- 172.21.15.103:0/90573792 >> 172.21.15.103:0/90573792 conn(0x7fa8200f71a0 msgr2=0x7fa8200f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.410+0000 7fa825a88700 1 -- 172.21.15.103:0/90573792 shutdown_connections 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.410+0000 7fa825a88700 1 -- 172.21.15.103:0/90573792 wait complete. 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.410+0000 7fa825a88700 1 Processor -- start 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.411+0000 7fa825a88700 1 -- start start 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.411+0000 7fa825a88700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 0x7fa8200fd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.411+0000 7fa825a88700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8200fb610 con 0x7fa8200fd700 2024-02-16T22:13:55.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.411+0000 7fa81f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 0x7fa8200fd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:55.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.411+0000 7fa81f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 0x7fa8200fd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:35194/0 (socket says 172.21.15.103:35194) 2024-02-16T22:13:55.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.411+0000 7fa81f7fe700 1 -- 172.21.15.103:0/1568807043 learned_addr learned my addr 172.21.15.103:0/1568807043 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:13:55.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.411+0000 7fa81f7fe700 1 -- 172.21.15.103:0/1568807043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa810005040 con 0x7fa8200fd700 2024-02-16T22:13:55.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.412+0000 7fa81f7fe700 1 --2- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 0x7fa8200fd190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa810009680 tx=0x7fa810009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:55.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.412+0000 7fa807fff700 1 -- 172.21.15.103:0/1568807043 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa81000b5c0 con 0x7fa8200fd700 2024-02-16T22:13:55.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.412+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8200fb810 con 0x7fa8200fd700 2024-02-16T22:13:55.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.412+0000 7fa807fff700 1 -- 172.21.15.103:0/1568807043 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa810004e60 con 0x7fa8200fd700 2024-02-16T22:13:55.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.412+0000 7fa807fff700 1 -- 172.21.15.103:0/1568807043 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa81001c590 con 0x7fa8200fd700 2024-02-16T22:13:55.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.412+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8200fbd00 con 0x7fa8200fd700 2024-02-16T22:13:55.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.413+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa820106990 con 0x7fa8200fd700 2024-02-16T22:13:55.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.413+0000 7fa807fff700 1 -- 172.21.15.103:0/1568807043 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7fa810013420 con 0x7fa8200fd700 2024-02-16T22:13:55.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.413+0000 7fa807fff700 1 --2- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fa808036870 0x7fa808038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:55.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.413+0000 7fa807fff700 1 -- 172.21.15.103:0/1568807043 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa810049b30 con 0x7fa8200fd700 2024-02-16T22:13:55.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.413+0000 7fa81effd700 1 --2- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fa808036870 0x7fa808038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:55.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.414+0000 7fa81effd700 1 --2- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fa808036870 0x7fa808038d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa814009990 tx=0x7fa814006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:55.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.418+0000 7fa807fff700 1 -- 172.21.15.103:0/1568807043 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa810025400 con 0x7fa8200fd700 2024-02-16T22:13:55.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.675+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa82005f5c0 con 0x7fa8200fd700 2024-02-16T22:13:55.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.676+0000 7fa807fff700 1 -- 172.21.15.103:0/1568807043 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fa810011370 con 0x7fa8200fd700 2024-02-16T22:13:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:13:55.678 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:13:55.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fa808036870 msgr2=0x7fa808038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:55.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 --2- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fa808036870 0x7fa808038d20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa814009990 tx=0x7fa814006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:55.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 msgr2=0x7fa8200fd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:55.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 --2- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 0x7fa8200fd190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa810009680 tx=0x7fa810009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:55.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 shutdown_connections 2024-02-16T22:13:55.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 --2- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fa808036870 0x7fa808038d20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:55.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 --2- 172.21.15.103:0/1568807043 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8200fd700 0x7fa8200fd190 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:55.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 >> 172.21.15.103:0/1568807043 conn(0x7fa8200f71a0 msgr2=0x7fa8200f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:55.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 shutdown_connections 2024-02-16T22:13:55.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:55.678+0000 7fa825a88700 1 -- 172.21.15.103:0/1568807043 wait complete. 2024-02-16T22:13:55.680 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:13:56.438 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:56 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/1568807043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:13:57.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:57 smithi002 ceph-mon[29154]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:13:57.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:13:57.254 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:13:57.500 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:57.547 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:13:59.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.048+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/844431127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 msgr2=0x7f5a280fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:59.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.048+0000 7f5a2cc1d700 1 --2- 172.21.15.103:0/844431127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 0x7f5a280fbee0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c0048c0 tx=0x7f5a1c009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:59.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.048+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/844431127 shutdown_connections 2024-02-16T22:13:59.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.048+0000 7f5a2cc1d700 1 --2- 172.21.15.103:0/844431127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 0x7f5a280fbee0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:59.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.048+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/844431127 >> 172.21.15.103:0/844431127 conn(0x7f5a280f7190 msgr2=0x7f5a280f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:59.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.048+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/844431127 shutdown_connections 2024-02-16T22:13:59.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.048+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/844431127 wait complete. 2024-02-16T22:13:59.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.049+0000 7f5a2cc1d700 1 Processor -- start 2024-02-16T22:13:59.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.049+0000 7f5a2cc1d700 1 -- start start 2024-02-16T22:13:59.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.049+0000 7f5a2cc1d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 0x7f5a2810be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:59.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.049+0000 7f5a2cc1d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2810c360 con 0x7f5a280fbad0 2024-02-16T22:13:59.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.049+0000 7f5a2659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 0x7f5a2810be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:59.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.049+0000 7f5a2659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 0x7f5a2810be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:55160/0 (socket says 172.21.15.103:55160) 2024-02-16T22:13:59.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.049+0000 7f5a2659c700 1 -- 172.21.15.103:0/70318712 learned_addr learned my addr 172.21.15.103:0/70318712 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:13:59.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.050+0000 7f5a2659c700 1 -- 172.21.15.103:0/70318712 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c005040 con 0x7f5a280fbad0 2024-02-16T22:13:59.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.050+0000 7f5a2659c700 1 --2- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 0x7f5a2810be20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009680 tx=0x7f5a1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:59.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.050+0000 7f5a16ffd700 1 -- 172.21.15.103:0/70318712 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a1c00bed0 con 0x7f5a280fbad0 2024-02-16T22:13:59.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.050+0000 7f5a16ffd700 1 -- 172.21.15.103:0/70318712 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a1c00b5c0 con 0x7f5a280fbad0 2024-02-16T22:13:59.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.050+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2810c560 con 0x7f5a280fbad0 2024-02-16T22:13:59.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.050+0000 7f5a16ffd700 1 -- 172.21.15.103:0/70318712 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a1c01c5b0 con 0x7f5a280fbad0 2024-02-16T22:13:59.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.050+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2810d120 con 0x7f5a280fbad0 2024-02-16T22:13:59.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.051+0000 7f5a16ffd700 1 -- 172.21.15.103:0/70318712 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 43423+0+0 (secure 0 0 0) 0x7f5a1c013420 con 0x7f5a280fbad0 2024-02-16T22:13:59.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.051+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2805f5c0 con 0x7f5a280fbad0 2024-02-16T22:13:59.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.051+0000 7f5a16ffd700 1 --2- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5a10036820 0x7f5a10038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:13:59.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.051+0000 7f5a16ffd700 1 -- 172.21.15.103:0/70318712 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5a1c049ad0 con 0x7f5a280fbad0 2024-02-16T22:13:59.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.051+0000 7f5a25d9b700 1 --2- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5a10036820 0x7f5a10038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:13:59.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.052+0000 7f5a25d9b700 1 --2- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5a10036820 0x7f5a10038cd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009990 tx=0x7f5a18006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:13:59.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.056+0000 7f5a16ffd700 1 -- 172.21.15.103:0/70318712 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5a1c025400 con 0x7f5a280fbad0 2024-02-16T22:13:59.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.309+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a2805f5c0 con 0x7f5a280fbad0 2024-02-16T22:13:59.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.309+0000 7f5a16ffd700 1 -- 172.21.15.103:0/70318712 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f5a1c004d30 con 0x7f5a280fbad0 2024-02-16T22:13:59.311 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:13:59.311 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:13:59.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.311+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5a10036820 msgr2=0x7f5a10038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:59.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.311+0000 7f5a2cc1d700 1 --2- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5a10036820 0x7f5a10038cd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009990 tx=0x7f5a18006e40 comp rx=0 tx=0).stop 2024-02-16T22:13:59.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.311+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 msgr2=0x7f5a2810be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:13:59.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.311+0000 7f5a2cc1d700 1 --2- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 0x7f5a2810be20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009680 tx=0x7f5a1c009e30 comp rx=0 tx=0).stop 2024-02-16T22:13:59.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.311+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 shutdown_connections 2024-02-16T22:13:59.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.311+0000 7f5a2cc1d700 1 --2- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f5a10036820 0x7f5a10038cd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:59.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.311+0000 7f5a2cc1d700 1 --2- 172.21.15.103:0/70318712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a280fbad0 0x7f5a2810be20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:13:59.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.312+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 >> 172.21.15.103:0/70318712 conn(0x7f5a280f7190 msgr2=0x7f5a280f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:13:59.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.312+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 shutdown_connections 2024-02-16T22:13:59.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:13:59.312+0000 7f5a2cc1d700 1 -- 172.21.15.103:0/70318712 wait complete. 2024-02-16T22:13:59.314 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:13:59.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:13:59 smithi002 ceph-mon[29154]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:00.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:00 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/70318712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:00.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:14:00.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:14:01.103 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:14:01.149 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:14:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:01 smithi002 ceph-mon[29154]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:01 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:01 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:01 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:01 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-16T22:14:02.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.568+0000 7fe5df778700 1 -- 172.21.15.103:0/1383807813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 msgr2=0x7fe5d80fbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:02.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.568+0000 7fe5df778700 1 --2- 172.21.15.103:0/1383807813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 0x7fe5d80fbef0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80048c0 tx=0x7fe5c8009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5df778700 1 -- 172.21.15.103:0/1383807813 shutdown_connections 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5df778700 1 --2- 172.21.15.103:0/1383807813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 0x7fe5d80fbef0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5df778700 1 -- 172.21.15.103:0/1383807813 >> 172.21.15.103:0/1383807813 conn(0x7fe5d80f71a0 msgr2=0x7fe5d80f95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5df778700 1 -- 172.21.15.103:0/1383807813 shutdown_connections 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5df778700 1 -- 172.21.15.103:0/1383807813 wait complete. 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5df778700 1 Processor -- start 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5df778700 1 -- start start 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5df778700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 0x7fe5d810fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5df778700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d80668d0 con 0x7fe5d80fbae0 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5dd514700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 0x7fe5d810fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5dd514700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 0x7fe5d810fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:55172/0 (socket says 172.21.15.103:55172) 2024-02-16T22:14:02.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.569+0000 7fe5dd514700 1 -- 172.21.15.103:0/3907120546 learned_addr learned my addr 172.21.15.103:0/3907120546 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:14:02.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.570+0000 7fe5dd514700 1 -- 172.21.15.103:0/3907120546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c8005040 con 0x7fe5d80fbae0 2024-02-16T22:14:02.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.570+0000 7fe5dd514700 1 --2- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 0x7fe5d810fbf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009680 tx=0x7fe5c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:02.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.570+0000 7fe5cdffb700 1 -- 172.21.15.103:0/3907120546 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5c800bed0 con 0x7fe5d80fbae0 2024-02-16T22:14:02.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.570+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d8066ad0 con 0x7fe5d80fbae0 2024-02-16T22:14:02.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.570+0000 7fe5cdffb700 1 -- 172.21.15.103:0/3907120546 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5c800b5c0 con 0x7fe5d80fbae0 2024-02-16T22:14:02.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.570+0000 7fe5cdffb700 1 -- 172.21.15.103:0/3907120546 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5c801c5b0 con 0x7fe5d80fbae0 2024-02-16T22:14:02.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.570+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d8110370 con 0x7fe5d80fbae0 2024-02-16T22:14:02.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.571+0000 7fe5cdffb700 1 -- 172.21.15.103:0/3907120546 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 14) v1 ==== 43437+0+0 (secure 0 0 0) 0x7fe5c8013420 con 0x7fe5d80fbae0 2024-02-16T22:14:02.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.571+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d81058b0 con 0x7fe5d80fbae0 2024-02-16T22:14:02.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.571+0000 7fe5cdffb700 1 --2- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe5c40368c0 0x7fe5c4038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:02.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.571+0000 7fe5cdffb700 1 -- 172.21.15.103:0/3907120546 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe5c8049bb0 con 0x7fe5d80fbae0 2024-02-16T22:14:02.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.571+0000 7fe5dcd13700 1 -- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe5c40368c0 msgr2=0x7fe5c4038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2481260931 2024-02-16T22:14:02.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.571+0000 7fe5dcd13700 1 --2- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe5c40368c0 0x7fe5c4038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-16T22:14:02.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.576+0000 7fe5cdffb700 1 -- 172.21.15.103:0/3907120546 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe5c8025400 con 0x7fe5d80fbae0 2024-02-16T22:14:02.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.772+0000 7fe5dcd13700 1 -- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe5c40368c0 msgr2=0x7fe5c4038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2481260931 2024-02-16T22:14:02.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.772+0000 7fe5dcd13700 1 --2- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe5c40368c0 0x7fe5c4038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-16T22:14:02.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.833+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5d804bed0 con 0x7fe5d80fbae0 2024-02-16T22:14:02.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.834+0000 7fe5cdffb700 1 -- 172.21.15.103:0/3907120546 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fe5c8004d30 con 0x7fe5d80fbae0 2024-02-16T22:14:02.836 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:14:02.836 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe5c40368c0 msgr2=0x7fe5c4038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 --2- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe5c40368c0 0x7fe5c4038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 msgr2=0x7fe5d810fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 --2- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 0x7fe5d810fbf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009680 tx=0x7fe5c8009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 shutdown_connections 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 --2- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7fe5c40368c0 0x7fe5c4038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 --2- 172.21.15.103:0/3907120546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d80fbae0 0x7fe5d810fbf0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 >> 172.21.15.103:0/3907120546 conn(0x7fe5d80f71a0 msgr2=0x7fe5d80f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 shutdown_connections 2024-02-16T22:14:02.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:02.836+0000 7fe5df778700 1 -- 172.21.15.103:0/3907120546 wait complete. 2024-02-16T22:14:02.839 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:14:03.090 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:02 smithi002 ceph-mon[29154]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-16T22:14:03.090 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:02 smithi002 ceph-mon[29154]: mgrmap e14: smithi002.resmjv(active, since 61s) 2024-02-16T22:14:03.948 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:03 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/3907120546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:04.409 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:14:04.410 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:14:04.662 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:14:04.711 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:14:06.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.021+0000 7f62315a8700 1 -- 172.21.15.103:0/222565300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 msgr2=0x7f622c0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.021+0000 7f62315a8700 1 --2- 172.21.15.103:0/222565300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 0x7f622c0fcfe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f621c0048c0 tx=0x7f621c009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.021+0000 7f62315a8700 1 -- 172.21.15.103:0/222565300 shutdown_connections 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.021+0000 7f62315a8700 1 --2- 172.21.15.103:0/222565300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 0x7f622c0fcfe0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.021+0000 7f62315a8700 1 -- 172.21.15.103:0/222565300 >> 172.21.15.103:0/222565300 conn(0x7f622c0f8270 msgr2=0x7f622c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.021+0000 7f62315a8700 1 -- 172.21.15.103:0/222565300 shutdown_connections 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.021+0000 7f62315a8700 1 -- 172.21.15.103:0/222565300 wait complete. 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.022+0000 7f62315a8700 1 Processor -- start 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.022+0000 7f62315a8700 1 -- start start 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.022+0000 7f62315a8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 0x7f622c10ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.022+0000 7f62315a8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c10d410 con 0x7f622c0fcbd0 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.022+0000 7f622affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 0x7f622c10ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.022+0000 7f622affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 0x7f622c10ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:56498/0 (socket says 172.21.15.103:56498) 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.022+0000 7f622affd700 1 -- 172.21.15.103:0/844100761 learned_addr learned my addr 172.21.15.103:0/844100761 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.022+0000 7f622affd700 1 -- 172.21.15.103:0/844100761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c005040 con 0x7f622c0fcbd0 2024-02-16T22:14:06.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.023+0000 7f622affd700 1 --2- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 0x7f622c10ced0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f621c009680 tx=0x7f621c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:06.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.023+0000 7f6223fff700 1 -- 172.21.15.103:0/844100761 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f621c00bed0 con 0x7f622c0fcbd0 2024-02-16T22:14:06.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.023+0000 7f6223fff700 1 -- 172.21.15.103:0/844100761 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f621c00b5c0 con 0x7f622c0fcbd0 2024-02-16T22:14:06.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.023+0000 7f6223fff700 1 -- 172.21.15.103:0/844100761 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f621c01c5b0 con 0x7f622c0fcbd0 2024-02-16T22:14:06.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.023+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622c10d610 con 0x7f622c0fcbd0 2024-02-16T22:14:06.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.023+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c10e2b0 con 0x7f622c0fcbd0 2024-02-16T22:14:06.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.024+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622c05f5c0 con 0x7f622c0fcbd0 2024-02-16T22:14:06.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.024+0000 7f6223fff700 1 -- 172.21.15.103:0/844100761 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 14) v1 ==== 43437+0+0 (secure 0 0 0) 0x7f621c013420 con 0x7f622c0fcbd0 2024-02-16T22:14:06.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.024+0000 7f6223fff700 1 --2- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f62180368c0 0x7f6218038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:06.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.024+0000 7f6223fff700 1 -- 172.21.15.103:0/844100761 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f621c049b00 con 0x7f622c0fcbd0 2024-02-16T22:14:06.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.024+0000 7f62227fc700 1 -- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f62180368c0 msgr2=0x7f6218038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2481260931 2024-02-16T22:14:06.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.024+0000 7f62227fc700 1 --2- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f62180368c0 0x7f6218038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-16T22:14:06.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.029+0000 7f6223fff700 1 -- 172.21.15.103:0/844100761 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f621c04be60 con 0x7f622c0fcbd0 2024-02-16T22:14:06.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.225+0000 7f62227fc700 1 -- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f62180368c0 msgr2=0x7f6218038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2481260931 2024-02-16T22:14:06.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.225+0000 7f62227fc700 1 --2- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f62180368c0 0x7f6218038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-16T22:14:06.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.286+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f622c10ee00 con 0x7f622c0fcbd0 2024-02-16T22:14:06.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.287+0000 7f6223fff700 1 -- 172.21.15.103:0/844100761 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f621c025400 con 0x7f622c0fcbd0 2024-02-16T22:14:06.289 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:14:06.289 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f62180368c0 msgr2=0x7f6218038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 --2- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f62180368c0 0x7f6218038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 msgr2=0x7f622c10ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 --2- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 0x7f622c10ced0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f621c009680 tx=0x7f621c009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 shutdown_connections 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 --2- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:6800/2481260931,v1:172.21.15.2:6801/2481260931] conn(0x7f62180368c0 0x7f6218038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 --2- 172.21.15.103:0/844100761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f622c0fcbd0 0x7f622c10ced0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 >> 172.21.15.103:0/844100761 conn(0x7f622c0f8270 msgr2=0x7f622c0f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 shutdown_connections 2024-02-16T22:14:06.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:06.289+0000 7f62315a8700 1 -- 172.21.15.103:0/844100761 wait complete. 2024-02-16T22:14:06.292 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:14:06.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:06 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/844100761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:07.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:14:07.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:14:08.053 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:14:08.100 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:14:08.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: Active manager daemon smithi002.resmjv restarted 2024-02-16T22:14:08.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: Activating manager daemon smithi002.resmjv 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: osdmap e5: 0 total, 0 up, 0 in 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: mgrmap e15: smithi002.resmjv(active, starting, since 0.00283731s) 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr metadata", "who": "smithi002.resmjv", "id": "smithi002.resmjv"}]: dispatch 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: Manager daemon smithi002.resmjv is now available 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:14:08.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:09.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.536+0000 7f910dc8e700 1 -- 172.21.15.103:0/385201108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 msgr2=0x7f91080fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:09.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.536+0000 7f910dc8e700 1 --2- 172.21.15.103:0/385201108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 0x7f91080fbed0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f90f80048c0 tx=0x7f90f8009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:09.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.536+0000 7f910dc8e700 1 -- 172.21.15.103:0/385201108 shutdown_connections 2024-02-16T22:14:09.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.536+0000 7f910dc8e700 1 --2- 172.21.15.103:0/385201108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 0x7f91080fbed0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:09.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.536+0000 7f910dc8e700 1 -- 172.21.15.103:0/385201108 >> 172.21.15.103:0/385201108 conn(0x7f91080f71a0 msgr2=0x7f91080f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:09.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.536+0000 7f910dc8e700 1 -- 172.21.15.103:0/385201108 shutdown_connections 2024-02-16T22:14:09.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.536+0000 7f910dc8e700 1 -- 172.21.15.103:0/385201108 wait complete. 2024-02-16T22:14:09.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.536+0000 7f910dc8e700 1 Processor -- start 2024-02-16T22:14:09.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.537+0000 7f910dc8e700 1 -- start start 2024-02-16T22:14:09.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.537+0000 7f910dc8e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 0x7f91080f9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:09.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.537+0000 7f910dc8e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91080f76b0 con 0x7f91080fbac0 2024-02-16T22:14:09.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.537+0000 7f91077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 0x7f91080f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.537+0000 7f91077fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 0x7f91080f9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:56526/0 (socket says 172.21.15.103:56526) 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.537+0000 7f91077fe700 1 -- 172.21.15.103:0/1737342506 learned_addr learned my addr 172.21.15.103:0/1737342506 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.537+0000 7f91077fe700 1 -- 172.21.15.103:0/1737342506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f8005040 con 0x7f91080fbac0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.537+0000 7f91077fe700 1 --2- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 0x7f91080f9230 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009b80 tx=0x7f90f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.538+0000 7f90effff700 1 -- 172.21.15.103:0/1737342506 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90f800b5c0 con 0x7f91080fbac0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.538+0000 7f90effff700 1 -- 172.21.15.103:0/1737342506 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90f800be70 con 0x7f91080fbac0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.538+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91080f78b0 con 0x7f91080fbac0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.538+0000 7f90effff700 1 -- 172.21.15.103:0/1737342506 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90f801c660 con 0x7f91080fbac0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.538+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91080f7da0 con 0x7f91080fbac0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.539+0000 7f90effff700 1 -- 172.21.15.103:0/1737342506 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f90f801c8d0 con 0x7f91080fbac0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.539+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91081068a0 con 0x7f91080fbac0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.539+0000 7f90effff700 1 --2- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f90f0036700 0x7f90f0038bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.539+0000 7f90effff700 1 -- 172.21.15.103:0/1737342506 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f90f8049a10 con 0x7f91080fbac0 2024-02-16T22:14:09.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.539+0000 7f9106ffd700 1 --2- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f90f0036700 0x7f90f0038bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:09.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.539+0000 7f9106ffd700 1 --2- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f90f0036700 0x7f90f0038bb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f90fc009990 tx=0x7f90fc006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:09.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.544+0000 7f90effff700 1 -- 172.21.15.103:0/1737342506 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f90f8023300 con 0x7f91080fbac0 2024-02-16T22:14:09.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:14:09.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/mirror_snapshot_schedule"}]: dispatch 2024-02-16T22:14:09.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/trash_purge_schedule"}]: dispatch 2024-02-16T22:14:09.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:09.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:09.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:09 smithi002 ceph-mon[29154]: mgrmap e16: smithi002.resmjv(active, since 1.0049s) 2024-02-16T22:14:09.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.813+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f910804bf20 con 0x7f91080fbac0 2024-02-16T22:14:09.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.813+0000 7f90effff700 1 -- 172.21.15.103:0/1737342506 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f90f8011b60 con 0x7f91080fbac0 2024-02-16T22:14:09.815 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:14:09.815 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:14:09.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.816+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f90f0036700 msgr2=0x7f90f0038bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:09.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.816+0000 7f910dc8e700 1 --2- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f90f0036700 0x7f90f0038bb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f90fc009990 tx=0x7f90fc006e40 comp rx=0 tx=0).stop 2024-02-16T22:14:09.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.816+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 msgr2=0x7f91080f9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:09.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.816+0000 7f910dc8e700 1 --2- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 0x7f91080f9230 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009b80 tx=0x7f90f8009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:09.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.816+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 shutdown_connections 2024-02-16T22:14:09.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.816+0000 7f910dc8e700 1 --2- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f90f0036700 0x7f90f0038bb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:09.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.816+0000 7f910dc8e700 1 --2- 172.21.15.103:0/1737342506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91080fbac0 0x7f91080f9230 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:09.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.816+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 >> 172.21.15.103:0/1737342506 conn(0x7f91080f71a0 msgr2=0x7f91080661d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:09.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.817+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 shutdown_connections 2024-02-16T22:14:09.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:09.817+0000 7f910dc8e700 1 -- 172.21.15.103:0/1737342506 wait complete. 2024-02-16T22:14:09.819 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:14:10.898 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:10 smithi002 ceph-mon[29154]: [16/Feb/2024:22:14:09] ENGINE Bus STARTING 2024-02-16T22:14:10.899 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:10 smithi002 ceph-mon[29154]: [16/Feb/2024:22:14:09] ENGINE Serving on https://172.21.15.2:7150 2024-02-16T22:14:10.899 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:10 smithi002 ceph-mon[29154]: [16/Feb/2024:22:14:09] ENGINE Bus STARTED 2024-02-16T22:14:10.899 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:10 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:10.899 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:10 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/1737342506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:10.899 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:10 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:11.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:14:11.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:14:11.847 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:14:12.186 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-16T22:14:12.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:12 smithi002 ceph-mon[29154]: mgrmap e17: smithi002.resmjv(active, since 2s) 2024-02-16T22:14:13.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:13.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:13.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:13.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:13.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:14:13.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:13.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:13.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:13.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:13.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:14:13.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:13.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:14:14.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.434+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2123263526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 msgr2=0x7ffaa40fe020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:14.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.434+0000 7ffaab4a3700 1 --2- 172.21.15.103:0/2123263526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 0x7ffaa40fe020 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ffaa00048c0 tx=0x7ffaa0009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:14.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.434+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2123263526 shutdown_connections 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.434+0000 7ffaab4a3700 1 --2- 172.21.15.103:0/2123263526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 0x7ffaa40fe020 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.434+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2123263526 >> 172.21.15.103:0/2123263526 conn(0x7ffaa4072850 msgr2=0x7ffaa4072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.434+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2123263526 shutdown_connections 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.434+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2123263526 wait complete. 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.435+0000 7ffaab4a3700 1 Processor -- start 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.435+0000 7ffaab4a3700 1 -- start start 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.435+0000 7ffaab4a3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 0x7ffaa4112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.435+0000 7ffaab4a3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa4113500 con 0x7ffaa40fdc50 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.435+0000 7ffaa923f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 0x7ffaa4112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.435+0000 7ffaa923f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 0x7ffaa4112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:56552/0 (socket says 172.21.15.103:56552) 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.435+0000 7ffaa923f700 1 -- 172.21.15.103:0/2864984731 learned_addr learned my addr 172.21.15.103:0/2864984731 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.435+0000 7ffaa923f700 1 -- 172.21.15.103:0/2864984731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa0005040 con 0x7ffaa40fdc50 2024-02-16T22:14:14.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.436+0000 7ffaa923f700 1 --2- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 0x7ffaa4112fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009300 tx=0x7ffaa0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:14.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.436+0000 7ffa99ffb700 1 -- 172.21.15.103:0/2864984731 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaa0004e60 con 0x7ffaa40fdc50 2024-02-16T22:14:14.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.436+0000 7ffa99ffb700 1 -- 172.21.15.103:0/2864984731 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffaa0013420 con 0x7ffaa40fdc50 2024-02-16T22:14:14.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.436+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa4113700 con 0x7ffaa40fdc50 2024-02-16T22:14:14.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.436+0000 7ffa99ffb700 1 -- 172.21.15.103:0/2864984731 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaa001c5b0 con 0x7ffaa40fdc50 2024-02-16T22:14:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.436+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa4113b70 con 0x7ffaa40fdc50 2024-02-16T22:14:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.437+0000 7ffa99ffb700 1 -- 172.21.15.103:0/2864984731 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7ffaa001c7f0 con 0x7ffaa40fdc50 2024-02-16T22:14:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.437+0000 7ffa99ffb700 1 --2- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffa90036960 0x7ffa90038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.437+0000 7ffa99ffb700 1 -- 172.21.15.103:0/2864984731 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ffaa0049c00 con 0x7ffaa40fdc50 2024-02-16T22:14:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.437+0000 7ffaa8a3e700 1 --2- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffa90036960 0x7ffa90038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.437+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa405f5c0 con 0x7ffaa40fdc50 2024-02-16T22:14:14.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.438+0000 7ffaa8a3e700 1 --2- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffa90036960 0x7ffa90038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009990 tx=0x7ffa94006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:14.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.442+0000 7ffa99ffb700 1 -- 172.21.15.103:0/2864984731 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ffaa0025920 con 0x7ffaa40fdc50 2024-02-16T22:14:14.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-16T22:14:14.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: Updating smithi002:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: Updating smithi103:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-16T22:14:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:14.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.730+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffaa4106f30 con 0x7ffaa40fdc50 2024-02-16T22:14:14.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.731+0000 7ffa99ffb700 1 -- 172.21.15.103:0/2864984731 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7ffaa0011b60 con 0x7ffaa40fdc50 2024-02-16T22:14:14.732 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:14:14.732 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffa90036960 msgr2=0x7ffa90038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 --2- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffa90036960 0x7ffa90038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009990 tx=0x7ffa94006e40 comp rx=0 tx=0).stop 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 msgr2=0x7ffaa4112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 --2- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 0x7ffaa4112fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009300 tx=0x7ffaa0009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 shutdown_connections 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 --2- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffa90036960 0x7ffa90038e10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 --2- 172.21.15.103:0/2864984731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaa40fdc50 0x7ffaa4112fc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 >> 172.21.15.103:0/2864984731 conn(0x7ffaa4072850 msgr2=0x7ffaa4101470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 shutdown_connections 2024-02-16T22:14:14.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:14.733+0000 7ffaab4a3700 1 -- 172.21.15.103:0/2864984731 wait complete. 2024-02-16T22:14:14.736 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:14:15.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:15 smithi002 ceph-mon[29154]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-16T22:14:15.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:15 smithi002 ceph-mon[29154]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-02-16T22:14:15.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:15 smithi002 ceph-mon[29154]: Updating smithi002:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.client.admin.keyring 2024-02-16T22:14:15.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:15 smithi002 ceph-mon[29154]: Updating smithi103:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.client.admin.keyring 2024-02-16T22:14:15.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:15 smithi002 ceph-mon[29154]: Deploying daemon crash.smithi103 on smithi103 2024-02-16T22:14:15.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:15 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/2864984731' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:16.766 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:14:16.766 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:14:17.023 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:18.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.703+0000 7f57c1348700 1 -- 172.21.15.103:0/3525047050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 msgr2=0x7f57bc0fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:18.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.703+0000 7f57c1348700 1 --2- 172.21.15.103:0/3525047050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 0x7f57bc0fbd90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0048c0 tx=0x7f57ac009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:18.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.703+0000 7f57c1348700 1 -- 172.21.15.103:0/3525047050 shutdown_connections 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.703+0000 7f57c1348700 1 --2- 172.21.15.103:0/3525047050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 0x7f57bc0fbd90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.703+0000 7f57c1348700 1 -- 172.21.15.103:0/3525047050 >> 172.21.15.103:0/3525047050 conn(0x7f57bc0f7640 msgr2=0x7f57bc0f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.703+0000 7f57c1348700 1 -- 172.21.15.103:0/3525047050 shutdown_connections 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.703+0000 7f57c1348700 1 -- 172.21.15.103:0/3525047050 wait complete. 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.704+0000 7f57c1348700 1 Processor -- start 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.704+0000 7f57c1348700 1 -- start start 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.704+0000 7f57c1348700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 0x7f57bc119610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.704+0000 7f57c1348700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc119b50 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.704+0000 7f57baffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 0x7f57bc119610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.704+0000 7f57baffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 0x7f57bc119610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:59692/0 (socket says 172.21.15.103:59692) 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.704+0000 7f57baffd700 1 -- 172.21.15.103:0/2030403088 learned_addr learned my addr 172.21.15.103:0/2030403088 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.705+0000 7f57baffd700 1 -- 172.21.15.103:0/2030403088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac005040 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.705+0000 7f57baffd700 1 --2- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 0x7f57bc119610 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009300 tx=0x7f57ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:18.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.705+0000 7f57a37fe700 1 -- 172.21.15.103:0/2030403088 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ac004e60 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.705+0000 7f57a37fe700 1 -- 172.21.15.103:0/2030403088 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57ac013420 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.705+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57bc119d50 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.705+0000 7f57a37fe700 1 -- 172.21.15.103:0/2030403088 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ac01c5b0 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.705+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc11a1c0 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.706+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57bc05f5c0 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.706+0000 7f57a37fe700 1 -- 172.21.15.103:0/2030403088 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7f57ac013a90 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.706+0000 7f57a37fe700 1 --2- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f57a4036960 0x7f57a4038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:18.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.706+0000 7f57a37fe700 1 -- 172.21.15.103:0/2030403088 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f57ac049b30 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.707+0000 7f57ba7fc700 1 --2- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f57a4036960 0x7f57a4038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:18.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.707+0000 7f57ba7fc700 1 --2- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f57a4036960 0x7f57a4038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009990 tx=0x7f57b0006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:18.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.712+0000 7f57a37fe700 1 -- 172.21.15.103:0/2030403088 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f57ac011960 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.978+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57bc0ffad0 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.979+0000 7f57a37fe700 1 -- 172.21.15.103:0/2030403088 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f57ac01b390 con 0x7f57bc0fb9c0 2024-02-16T22:14:18.981 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:14:18.981 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:14:18.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f57a4036960 msgr2=0x7f57a4038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:18.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 --2- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f57a4036960 0x7f57a4038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009990 tx=0x7f57b0006e40 comp rx=0 tx=0).stop 2024-02-16T22:14:18.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 msgr2=0x7f57bc119610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:18.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 --2- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 0x7f57bc119610 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009300 tx=0x7f57ac009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:18.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 shutdown_connections 2024-02-16T22:14:18.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 --2- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f57a4036960 0x7f57a4038e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:18.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 --2- 172.21.15.103:0/2030403088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57bc0fb9c0 0x7f57bc119610 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:18.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 >> 172.21.15.103:0/2030403088 conn(0x7f57bc0f7640 msgr2=0x7f57bc0f82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:18.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 shutdown_connections 2024-02-16T22:14:18.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:18.981+0000 7f57c1348700 1 -- 172.21.15.103:0/2030403088 wait complete. 2024-02-16T22:14:18.984 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:14:20.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:19 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:20.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:19 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:20.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:19 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:20.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:19 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:20.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:19 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/2030403088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:20.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:14:20.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:14:20.817 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:21.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:20 smithi002 ceph-mon[29154]: Deploying daemon node-exporter.smithi103 on smithi103 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.898+0000 7f40de5bf700 1 -- 172.21.15.103:0/2802855893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 msgr2=0x7f40d8101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.898+0000 7f40de5bf700 1 --2- 172.21.15.103:0/2802855893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 0x7f40d8101f10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f40c80048c0 tx=0x7f40c8009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.898+0000 7f40de5bf700 1 -- 172.21.15.103:0/2802855893 shutdown_connections 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.898+0000 7f40de5bf700 1 --2- 172.21.15.103:0/2802855893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 0x7f40d8101f10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.898+0000 7f40de5bf700 1 -- 172.21.15.103:0/2802855893 >> 172.21.15.103:0/2802855893 conn(0x7f40d80f7660 msgr2=0x7f40d80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.898+0000 7f40de5bf700 1 -- 172.21.15.103:0/2802855893 shutdown_connections 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.898+0000 7f40de5bf700 1 -- 172.21.15.103:0/2802855893 wait complete. 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.899+0000 7f40de5bf700 1 Processor -- start 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.899+0000 7f40de5bf700 1 -- start start 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.899+0000 7f40de5bf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 0x7f40d810dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:23.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.899+0000 7f40de5bf700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d810e510 con 0x7f40d8101b40 2024-02-16T22:14:23.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.900+0000 7f40d7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 0x7f40d810dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:23.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.900+0000 7f40d7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 0x7f40d810dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:59704/0 (socket says 172.21.15.103:59704) 2024-02-16T22:14:23.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.900+0000 7f40d7fff700 1 -- 172.21.15.103:0/3832944856 learned_addr learned my addr 172.21.15.103:0/3832944856 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:14:23.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.900+0000 7f40d7fff700 1 -- 172.21.15.103:0/3832944856 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c8005040 con 0x7f40d8101b40 2024-02-16T22:14:23.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.900+0000 7f40d7fff700 1 --2- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 0x7f40d810dfd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009300 tx=0x7f40c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:23.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.900+0000 7f40d4ff9700 1 -- 172.21.15.103:0/3832944856 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c8004e60 con 0x7f40d8101b40 2024-02-16T22:14:23.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.900+0000 7f40d4ff9700 1 -- 172.21.15.103:0/3832944856 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40c8013420 con 0x7f40d8101b40 2024-02-16T22:14:23.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.900+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d810e710 con 0x7f40d8101b40 2024-02-16T22:14:23.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.900+0000 7f40d4ff9700 1 -- 172.21.15.103:0/3832944856 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40c801c590 con 0x7f40d8101b40 2024-02-16T22:14:23.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.901+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d810ec00 con 0x7f40d8101b40 2024-02-16T22:14:23.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.901+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d80ffd50 con 0x7f40d8101b40 2024-02-16T22:14:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.901+0000 7f40d4ff9700 1 -- 172.21.15.103:0/3832944856 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7f40c801c850 con 0x7f40d8101b40 2024-02-16T22:14:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.902+0000 7f40d4ff9700 1 --2- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f40c0036910 0x7f40c0038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.902+0000 7f40d4ff9700 1 -- 172.21.15.103:0/3832944856 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f40c8049c80 con 0x7f40d8101b40 2024-02-16T22:14:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.902+0000 7f40d77fe700 1 --2- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f40c0036910 0x7f40c0038dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:23.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.902+0000 7f40d77fe700 1 --2- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f40c0036910 0x7f40c0038dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009990 tx=0x7f40cc006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:23.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:23.907+0000 7f40d4ff9700 1 -- 172.21.15.103:0/3832944856 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f40c8050080 con 0x7f40d8101b40 2024-02-16T22:14:24.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.175+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40d804bed0 con 0x7f40d8101b40 2024-02-16T22:14:24.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.176+0000 7f40d4ff9700 1 -- 172.21.15.103:0/3832944856 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f40c8023300 con 0x7f40d8101b40 2024-02-16T22:14:24.177 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:14:24.177 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:14:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f40c0036910 msgr2=0x7f40c0038dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 --2- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f40c0036910 0x7f40c0038dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009990 tx=0x7f40cc006e40 comp rx=0 tx=0).stop 2024-02-16T22:14:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 msgr2=0x7f40d810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 --2- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 0x7f40d810dfd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009300 tx=0x7f40c8009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 shutdown_connections 2024-02-16T22:14:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 --2- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f40c0036910 0x7f40c0038dc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 --2- 172.21.15.103:0/3832944856 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40d8101b40 0x7f40d810dfd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 >> 172.21.15.103:0/3832944856 conn(0x7f40d80f7660 msgr2=0x7f40d8103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:24.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 shutdown_connections 2024-02-16T22:14:24.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:24.178+0000 7f40de5bf700 1 -- 172.21.15.103:0/3832944856 wait complete. 2024-02-16T22:14:24.181 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.ydydbo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.ydydbo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: Deploying daemon mgr.smithi103.ydydbo on smithi103 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:14:24.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:24 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/3832944856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:26.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:14:26.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:14:26.736 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:28.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:28.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:28.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:28.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:28.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-16T22:14:28.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:28.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:27 smithi002 ceph-mon[29154]: Deploying daemon mon.smithi103 on smithi103 2024-02-16T22:14:29.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.210+0000 7f54f507a700 1 -- 172.21.15.103:0/2817172644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 msgr2=0x7f54f0102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:29.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.210+0000 7f54f507a700 1 --2- 172.21.15.103:0/2817172644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 0x7f54f0102fa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f54e40048c0 tx=0x7f54e4009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:29.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.210+0000 7f54f507a700 1 -- 172.21.15.103:0/2817172644 shutdown_connections 2024-02-16T22:14:29.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.210+0000 7f54f507a700 1 --2- 172.21.15.103:0/2817172644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 0x7f54f0102fa0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:29.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.210+0000 7f54f507a700 1 -- 172.21.15.103:0/2817172644 >> 172.21.15.103:0/2817172644 conn(0x7f54f00f8760 msgr2=0x7f54f00fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:29.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.210+0000 7f54f507a700 1 -- 172.21.15.103:0/2817172644 shutdown_connections 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.210+0000 7f54f507a700 1 -- 172.21.15.103:0/2817172644 wait complete. 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.211+0000 7f54f507a700 1 Processor -- start 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.211+0000 7f54f507a700 1 -- start start 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.211+0000 7f54f507a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 0x7f54f0112f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.211+0000 7f54f507a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0113470 con 0x7f54f0102bd0 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.211+0000 7f54eed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 0x7f54f0112f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.211+0000 7f54eed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 0x7f54f0112f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.103:35382/0 (socket says 172.21.15.103:35382) 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.211+0000 7f54eed9d700 1 -- 172.21.15.103:0/2226930013 learned_addr learned my addr 172.21.15.103:0/2226930013 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.211+0000 7f54eed9d700 1 -- 172.21.15.103:0/2226930013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e4005040 con 0x7f54f0102bd0 2024-02-16T22:14:29.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.212+0000 7f54eed9d700 1 --2- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 0x7f54f0112f30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54e4009300 tx=0x7f54e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:29.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.212+0000 7f54d77fe700 1 -- 172.21.15.103:0/2226930013 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54e4004e60 con 0x7f54f0102bd0 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.212+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f0113670 con 0x7f54f0102bd0 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.212+0000 7f54d77fe700 1 -- 172.21.15.103:0/2226930013 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54e4013420 con 0x7f54f0102bd0 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.212+0000 7f54d77fe700 1 -- 172.21.15.103:0/2226930013 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54e401c5b0 con 0x7f54f0102bd0 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.212+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f0113b60 con 0x7f54f0102bd0 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.213+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f005f5c0 con 0x7f54f0102bd0 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.213+0000 7f54d77fe700 1 -- 172.21.15.103:0/2226930013 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 43479+0+0 (secure 0 0 0) 0x7f54e401c7f0 con 0x7f54f0102bd0 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.213+0000 7f54d77fe700 1 --2- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f54d8036960 0x7f54d8038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.213+0000 7f54d77fe700 1 -- 172.21.15.103:0/2226930013 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f54e4049b80 con 0x7f54f0102bd0 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.213+0000 7f54ee59c700 1 --2- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f54d8036960 0x7f54d8038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:29.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.214+0000 7f54ee59c700 1 --2- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f54d8036960 0x7f54d8038e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54e0009990 tx=0x7f54e0006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:29.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.218+0000 7f54d77fe700 1 -- 172.21.15.103:0/2226930013 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f54e4025920 con 0x7f54f0102bd0 2024-02-16T22:14:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.520+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54f010e6e0 con 0x7f54f0102bd0 2024-02-16T22:14:29.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.520+0000 7f54d77fe700 1 -- 172.21.15.103:0/2226930013 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f54e4011b60 con 0x7f54f0102bd0 2024-02-16T22:14:29.523 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:14:29.523 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:11:56.420723Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-16T22:14:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.522+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f54d8036960 msgr2=0x7f54d8038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.522+0000 7f54f507a700 1 --2- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f54d8036960 0x7f54d8038e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54e0009990 tx=0x7f54e0006e40 comp rx=0 tx=0).stop 2024-02-16T22:14:29.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.522+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 msgr2=0x7f54f0112f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.522+0000 7f54f507a700 1 --2- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 0x7f54f0112f30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54e4009300 tx=0x7f54e4009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.522+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 shutdown_connections 2024-02-16T22:14:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.522+0000 7f54f507a700 1 --2- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f54d8036960 0x7f54d8038e10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.522+0000 7f54f507a700 1 --2- 172.21.15.103:0/2226930013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54f0102bd0 0x7f54f0112f30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.522+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 >> 172.21.15.103:0/2226930013 conn(0x7f54f00f8760 msgr2=0x7f54f00f93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.523+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 shutdown_connections 2024-02-16T22:14:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:29.523+0000 7f54f507a700 1 -- 172.21.15.103:0/2226930013 wait complete. 2024-02-16T22:14:29.526 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-02-16T22:14:29.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:29 smithi002 ceph-mon[29154]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:29.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:29 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:29.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:29 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/2226930013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:31.187 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-16T22:14:31.188 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mon dump -f json 2024-02-16T22:14:31.481 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi103/config 2024-02-16T22:14:32.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:31 smithi002 ceph-mon[29154]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.214 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 podman[36231]: 2024-02-16 22:14:33.193673081 +0000 UTC m=+0.507927854 container start 3d198bee2a92b0107c293b6b76f2731561963318762a2dbdf6086c11dc848f54 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi103, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-b88fae6, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-16T22:14:33.493 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: set uid:gid to 167:167 (ceph:ceph) 2024-02-16T22:14:33.493 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable), process ceph-mon, pid 2 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pidfile_write: ignore empty --pid-file 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: load: jerasure load: lrc 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: RocksDB version: 6.15.5 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Compile date Feb 15 2024 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: DB SUMMARY 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: DB Session ID: W60IB6P0P36D9CXHOIED 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: CURRENT file: CURRENT 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: IDENTITY file: IDENTITY 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi103/store.db dir, Total Num: 0, files: 2024-02-16T22:14:33.494 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi103/store.db: 000004.log size: 511 ; 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.error_if_exists: 0 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.create_if_missing: 0 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.paranoid_checks: 1 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.env: 0x55c89d827a80 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.fs: Posix File System 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.info_log: 0x55c89f53dcc0 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_file_opening_threads: 16 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.statistics: (nil) 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.use_fsync: 0 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_log_file_size: 0 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-16T22:14:33.495 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.keep_log_file_num: 1000 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.recycle_log_file_num: 0 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.allow_fallocate: 1 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.allow_mmap_reads: 0 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.allow_mmap_writes: 0 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.use_direct_reads: 0 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.create_missing_column_families: 0 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.db_log_dir: 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi103/store.db 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-16T22:14:33.496 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.advise_random_on_open: 1 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.db_write_buffer_size: 0 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.write_buffer_manager: 0x55c8a02b2b40 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.rate_limiter: (nil) 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.wal_recovery_mode: 2 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.enable_thread_tracking: 0 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.enable_pipelined_write: 0 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.unordered_write: 0 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-16T22:14:33.497 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.row_cache: None 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.wal_filter: None 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.allow_ingest_behind: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.preserve_deletes: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.two_write_queues: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.manual_wal_flush: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.atomic_flush: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.log_readahead_size: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.best_efforts_recovery: 0 2024-02-16T22:14:33.498 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.allow_data_in_errors: 0 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.db_host_id: __hostname__ 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_background_jobs: 2 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_background_compactions: -1 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_subcompactions: 1 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_total_wal_size: 0 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_open_files: -1 2024-02-16T22:14:33.499 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bytes_per_sync: 0 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_readahead_size: 0 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_background_flushes: -1 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Compression algorithms supported: 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: kZSTD supported: 0 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: kXpressCompression supported: 0 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: kLZ4HCCompression supported: 1 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: kZlibCompression supported: 1 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: kSnappyCompression supported: 1 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: kLZ4Compression supported: 1 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: kBZip2Compression supported: 0 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-16T22:14:33.500 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi103/store.db/MANIFEST-000003 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.merge_operator: 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_filter: None 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_filter_factory: None 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.sst_partitioner_factory: None 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c89f592f58) 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: cache_index_and_filter_blocks: 1 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: pin_top_level_index_and_filter: 1 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: index_type: 0 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: data_block_index_type: 0 2024-02-16T22:14:33.501 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: index_shortening: 1 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: hash_index_allow_collision: 1 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: checksum: 1 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: no_block_cache: 0 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache: 0x55c89f5c5090 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache_name: BinnedLRUCache 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache_options: 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: capacity : 536870912 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: num_shard_bits : 4 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: strict_capacity_limit : 0 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: high_pri_pool_ratio: 0.000 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_cache_compressed: (nil) 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: persistent_cache: (nil) 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_size: 4096 2024-02-16T22:14:33.502 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_size_deviation: 10 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_restart_interval: 16 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: index_block_restart_interval: 1 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: metadata_block_size: 4096 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: partition_filters: 0 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: use_delta_encoding: 1 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: whole_key_filtering: 1 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: verify_compression: 0 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: read_amp_bytes_per_bit: 0 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: format_version: 4 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: enable_index_compression: 1 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: block_align: 0 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.write_buffer_size: 33554432 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_write_buffer_number: 2 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compression: NoCompression 2024-02-16T22:14:33.503 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bottommost_compression: Disabled 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.prefix_extractor: nullptr 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.num_levels: 7 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-16T22:14:33.504 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compression_opts.level: 32767 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compression_opts.strategy: 0 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compression_opts.enabled: false 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.target_file_size_base: 67108864 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-16T22:14:33.505 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.arena_block_size: 4194304 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.disable_auto_compactions: 0 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-16T22:14:33.506 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.table_properties_collectors: 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.inplace_update_support: 0 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.bloom_locality: 0 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.max_successive_merges: 0 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.paranoid_file_checks: 0 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.force_consistency_checks: 1 2024-02-16T22:14:33.507 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.report_bg_io_stats: 0 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.ttl: 2592000 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.enable_blob_files: false 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.min_blob_size: 0 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.blob_file_size: 268435456 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi103/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708121673219250, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-16T22:14:33.508 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708121673219551, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708121673, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "21a29870-7258-4559-8d7d-ebf18eed676a", "db_session_id": "W60IB6P0P36D9CXHOIED"}} 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708121673219861, "job": 1, "event": "recovery_finished"} 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi103/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55c89f559180 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: DB pointer 0x55c89f62e000 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** DB Stats ** 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-16T22:14:33.509 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-16T22:14:33.510 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** Compaction Stats [default] ** 2024-02-16T22:14:33.511 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-16T22:14:33.512 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103 does not exist in monmap, will attempt to join an existing cluster 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: using public_addr v2:172.21.15.103:0/0 -> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: starting mon.smithi103 rank -1 at public addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] at bind addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi103 fsid 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(???) e0 preinit fsid 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).mds e1 new map 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).mds e1 print_map 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: e1 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: legacy client fscid: -1 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout: No filesystems configured 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-16T22:14:33.513 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Deploying daemon node-exporter.smithi002 on smithi002 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:14:33.514 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Deploying cephadm binary to smithi103 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Added host smithi103 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Deploying daemon alertmanager.smithi002 on smithi002 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/839973598' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.515 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/839973598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: osdmap e4: 0 total, 0 up, 0 in 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi002:172.21.15.2=smithi002;smithi103:172.21.15.103=smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Saving service mon spec with placement smithi002:172.21.15.2=smithi002;smithi103:172.21.15.103=smithi103;count:2 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-16T22:14:33.516 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Deploying daemon grafana.smithi002 on smithi002 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/6933436' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/3964559962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/4207948971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/4011184044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.517 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Deploying daemon prometheus.smithi002 on smithi002 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/4293791560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/259101024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/1568807043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.518 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.519 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.519 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/70318712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.519 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.519 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.519 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.519 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.519 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14162 172.21.15.2:0/1552821399' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mgrmap e14: smithi002.resmjv(active, since 61s) 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/3907120546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/844100761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Active manager daemon smithi002.resmjv restarted 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Activating manager daemon smithi002.resmjv 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: osdmap e5: 0 total, 0 up, 0 in 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mgrmap e15: smithi002.resmjv(active, starting, since 0.00283731s) 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr metadata", "who": "smithi002.resmjv", "id": "smithi002.resmjv"}]: dispatch 2024-02-16T22:14:33.520 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Manager daemon smithi002.resmjv is now available 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/mirror_snapshot_schedule"}]: dispatch 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi002.resmjv/trash_purge_schedule"}]: dispatch 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mgrmap e16: smithi002.resmjv(active, since 1.0049s) 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: [16/Feb/2024:22:14:09] ENGINE Bus STARTING 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: [16/Feb/2024:22:14:09] ENGINE Serving on https://172.21.15.2:7150 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: [16/Feb/2024:22:14:09] ENGINE Bus STARTED 2024-02-16T22:14:33.521 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/1737342506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mgrmap e17: smithi002.resmjv(active, since 2s) 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:14:33.522 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Updating smithi002:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Updating smithi103:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:33.523 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Updating smithi002:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.client.admin.keyring 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Updating smithi103:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.client.admin.keyring 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Deploying daemon crash.smithi103 on smithi103 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/2864984731' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/2030403088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Deploying daemon node-exporter.smithi103 on smithi103 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.524 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.ydydbo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.ydydbo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Deploying daemon mgr.smithi103.ydydbo on smithi103 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/3832944856' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.525 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: Deploying daemon mon.smithi103 on smithi103 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/2226930013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 ceph-mon[36259]: mon.smithi103@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 bash[36231]: 3d198bee2a92b0107c293b6b76f2731561963318762a2dbdf6086c11dc848f54 2024-02-16T22:14:33.526 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:33 smithi103 systemd[1]: Started Ceph mon.smithi103 for 43c11c36-cd18-11ee-95ba-87774f69a715. 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: mon.smithi002 calling monitor election 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.? 172.21.15.103:0/1547805569' entity='mgr.smithi103.ydydbo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.ydydbo/crt"}]: dispatch 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: mon.smithi103 calling monitor election 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: mon.smithi002 is new leader, mons smithi002,smithi103 in quorum (ranks 0,1) 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: monmap e2: 2 mons at {smithi002=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],smithi103=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: fsmap 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: osdmap e5: 0 total, 0 up, 0 in 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: mgrmap e17: smithi002.resmjv(active, since 29s) 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: Standby manager daemon smithi103.ydydbo started 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.? 172.21.15.103:0/1547805569' entity='mgr.smithi103.ydydbo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: overall HEALTH_OK 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.? 172.21.15.103:0/1547805569' entity='mgr.smithi103.ydydbo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.ydydbo/key"}]: dispatch 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.? 172.21.15.103:0/1547805569' entity='mgr.smithi103.ydydbo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-16T22:14:38.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:38.728 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:38.728 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:38.728 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:38.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:14:38.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: mon.smithi002 calling monitor election 2024-02-16T22:14:38.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.? 172.21.15.103:0/1547805569' entity='mgr.smithi103.ydydbo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.ydydbo/crt"}]: dispatch 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: mon.smithi103 calling monitor election 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: mon.smithi002 is new leader, mons smithi002,smithi103 in quorum (ranks 0,1) 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: monmap e2: 2 mons at {smithi002=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],smithi103=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: fsmap 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: osdmap e5: 0 total, 0 up, 0 in 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: mgrmap e17: smithi002.resmjv(active, since 29s) 2024-02-16T22:14:38.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: Standby manager daemon smithi103.ydydbo started 2024-02-16T22:14:38.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.? 172.21.15.103:0/1547805569' entity='mgr.smithi103.ydydbo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-16T22:14:38.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: overall HEALTH_OK 2024-02-16T22:14:38.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.? 172.21.15.103:0/1547805569' entity='mgr.smithi103.ydydbo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi103.ydydbo/key"}]: dispatch 2024-02-16T22:14:38.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:38.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.? 172.21.15.103:0/1547805569' entity='mgr.smithi103.ydydbo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-16T22:14:38.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:38.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:38.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:38.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:39.597 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:39 smithi103 ceph-mon[36259]: mgrmap e18: smithi002.resmjv(active, since 29s), standbys: smithi103.ydydbo 2024-02-16T22:14:39.597 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:39 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr metadata", "who": "smithi103.ydydbo", "id": "smithi103.ydydbo"}]: dispatch 2024-02-16T22:14:39.597 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:39 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:14:39.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:39 smithi103 ceph-mon[36259]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:39.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:39 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:14:39.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:39 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:39.598 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:39 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:39.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.714+0000 7fd4f6956700 1 -- 172.21.15.103:0/465053217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd4d8003580 msgr2=0x7fd4d8005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:39.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.714+0000 7fd4f6956700 1 --2- 172.21.15.103:0/465053217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd4d8003580 0x7fd4d8005a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0065630 tx=0x7fd4e4002040 comp rx=0 tx=0).stop 2024-02-16T22:14:39.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.714+0000 7fd4f6956700 1 -- 172.21.15.103:0/465053217 shutdown_connections 2024-02-16T22:14:39.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.714+0000 7fd4f6956700 1 --2- 172.21.15.103:0/465053217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd4d8003580 0x7fd4d8005a00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:39.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.714+0000 7fd4f6956700 1 -- 172.21.15.103:0/465053217 >> 172.21.15.103:0/465053217 conn(0x7fd4f00729f0 msgr2=0x7fd4f0072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:39.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.714+0000 7fd4f6956700 1 -- 172.21.15.103:0/465053217 shutdown_connections 2024-02-16T22:14:39.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.714+0000 7fd4f6956700 1 -- 172.21.15.103:0/465053217 wait complete. 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.715+0000 7fd4f6956700 1 Processor -- start 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.715+0000 7fd4f6956700 1 -- start start 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4f6956700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd4d8003580 0x7fd4f01132a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4f6956700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 0x7fd4f010d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4f6956700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0113df0 con 0x7fd4d8003580 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4f6956700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f010d680 con 0x7fd4f01137e0 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4ef7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 0x7fd4f010d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4ef7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 0x7fd4f010d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37636/0 (socket says 172.21.15.103:37636) 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4ef7fe700 1 -- 172.21.15.103:0/4028307639 learned_addr learned my addr 172.21.15.103:0/4028307639 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4ef7fe700 1 -- 172.21.15.103:0/4028307639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 msgr2=0x7fd4f010d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4ef7fe700 1 -- 172.21.15.103:0/4028307639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 msgr2=0x7fd4f010d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4ef7fe700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 0x7fd4f010d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-16T22:14:39.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4ef7fe700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 0x7fd4f010d140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4effff700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd4d8003580 0x7fd4f01132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4effff700 1 -- 172.21.15.103:0/4028307639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 msgr2=0x7fd4f010d140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4effff700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 0x7fd4f010d140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.716+0000 7fd4effff700 1 -- 172.21.15.103:0/4028307639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e4005040 con 0x7fd4d8003580 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.717+0000 7fd4effff700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd4d8003580 0x7fd4f01132a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4007e30 tx=0x7fd4e4002040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.717+0000 7fd4ecff9700 1 -- 172.21.15.103:0/4028307639 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4e4009260 con 0x7fd4d8003580 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.717+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f010d900 con 0x7fd4d8003580 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.717+0000 7fd4ecff9700 1 -- 172.21.15.103:0/4028307639 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4e4009890 con 0x7fd4d8003580 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.717+0000 7fd4ecff9700 1 -- 172.21.15.103:0/4028307639 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4e4022bc0 con 0x7fd4d8003580 2024-02-16T22:14:39.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.717+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f010dde0 con 0x7fd4d8003580 2024-02-16T22:14:39.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.718+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f005f5c0 con 0x7fd4d8003580 2024-02-16T22:14:39.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.718+0000 7fd4ecff9700 1 -- 172.21.15.103:0/4028307639 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7fd4e40093c0 con 0x7fd4d8003580 2024-02-16T22:14:39.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.719+0000 7fd4ecff9700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fd4dc069320 0x7fd4dc06b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:39.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.719+0000 7fd4ecff9700 1 -- 172.21.15.103:0/4028307639 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd4e40886e0 con 0x7fd4d8003580 2024-02-16T22:14:39.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.719+0000 7fd4ef7fe700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fd4dc069320 0x7fd4dc06b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:39.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.719+0000 7fd4ef7fe700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fd4dc069320 0x7fd4dc06b7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0006fd0 tx=0x7fd4e0008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:39.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:39 smithi002 ceph-mon[29154]: mgrmap e18: smithi002.resmjv(active, since 29s), standbys: smithi103.ydydbo 2024-02-16T22:14:39.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:39 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr metadata", "who": "smithi103.ydydbo", "id": "smithi103.ydydbo"}]: dispatch 2024-02-16T22:14:39.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:39 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:14:39.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:39 smithi002 ceph-mon[29154]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:39.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:39 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:14:39.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:39 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:39.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:39 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:14:39.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:39.724+0000 7fd4ecff9700 1 -- 172.21.15.103:0/4028307639 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd4e40594a0 con 0x7fd4d8003580 2024-02-16T22:14:40.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.002+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4f0104a00 con 0x7fd4d8003580 2024-02-16T22:14:40.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.003+0000 7fd4ecff9700 1 -- 172.21.15.103:0/4028307639 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1013 (secure 0 0 0) 0x7fd4f0104a00 con 0x7fd4d8003580 2024-02-16T22:14:40.005 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-02-16T22:14:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":2,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","modified":"2024-02-16T22:14:33.247877Z","created":"2024-02-16T22:11:56.420723Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi002","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi103","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-16T22:14:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fd4dc069320 msgr2=0x7fd4dc06b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fd4dc069320 0x7fd4dc06b7d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0006fd0 tx=0x7fd4e0008040 comp rx=0 tx=0).stop 2024-02-16T22:14:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd4d8003580 msgr2=0x7fd4f01132a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd4d8003580 0x7fd4f01132a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4007e30 tx=0x7fd4e4002040 comp rx=0 tx=0).stop 2024-02-16T22:14:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 shutdown_connections 2024-02-16T22:14:40.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fd4dc069320 0x7fd4dc06b7d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:40.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd4d8003580 0x7fd4f01132a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:40.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 --2- 172.21.15.103:0/4028307639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4f01137e0 0x7fd4f010d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:40.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 >> 172.21.15.103:0/4028307639 conn(0x7fd4f00729f0 msgr2=0x7fd4f00fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:40.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 shutdown_connections 2024-02-16T22:14:40.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:14:40.005+0000 7fd4f6956700 1 -- 172.21.15.103:0/4028307639 wait complete. 2024-02-16T22:14:40.008 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 2 2024-02-16T22:14:40.978 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-16T22:14:40.979 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph config generate-minimal-conf 2024-02-16T22:14:41.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:40 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/4028307639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:41.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:40 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:41.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:40 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:41.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:40 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:41.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:40 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:14:41.244 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:14:41.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:40 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/4028307639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-16T22:14:41.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:40 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:41.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:40 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:41.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:40 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:41.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:40 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:14:41.967 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-16T22:14:41.967 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-16T22:14:41.967 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:41.967 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: Updating smithi002:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:41.967 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: Updating smithi103:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:41.967 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:41.967 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:41.968 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:41.968 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:41.968 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:41.968 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-16T22:14:41.968 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-16T22:14:41.968 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:41 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:42.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: Updating smithi002:/etc/ceph/ceph.conf 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: Updating smithi103:/etc/ceph/ceph.conf 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: Updating smithi002:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: Updating smithi103:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/config/ceph.conf 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-16T22:14:42.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:41 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:43.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:42 smithi002 ceph-mon[29154]: Reconfiguring mon.smithi002 (unknown last config time)... 2024-02-16T22:14:43.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:42 smithi002 ceph-mon[29154]: Reconfiguring daemon mon.smithi002 on smithi002 2024-02-16T22:14:43.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:42 smithi103 ceph-mon[36259]: Reconfiguring mon.smithi002 (unknown last config time)... 2024-02-16T22:14:43.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:42 smithi103 ceph-mon[36259]: Reconfiguring daemon mon.smithi002 on smithi002 2024-02-16T22:14:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.293+0000 7f6a1baa8700 1 -- 172.21.15.2:0/3549600925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a14102bb0 msgr2=0x7f6a14102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.293+0000 7f6a1baa8700 1 --2- 172.21.15.2:0/3549600925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a14102bb0 0x7f6a14102f80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a100048c0 tx=0x7f6a10009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.293+0000 7f6a1baa8700 1 -- 172.21.15.2:0/3549600925 shutdown_connections 2024-02-16T22:14:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.293+0000 7f6a1baa8700 1 --2- 172.21.15.2:0/3549600925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a14102bb0 0x7f6a14102f80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.293+0000 7f6a1baa8700 1 -- 172.21.15.2:0/3549600925 >> 172.21.15.2:0/3549600925 conn(0x7f6a140f8760 msgr2=0x7f6a140fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.293+0000 7f6a1baa8700 1 -- 172.21.15.2:0/3549600925 shutdown_connections 2024-02-16T22:14:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a1baa8700 1 -- 172.21.15.2:0/3549600925 wait complete. 2024-02-16T22:14:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a1baa8700 1 Processor -- start 2024-02-16T22:14:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a1baa8700 1 -- start start 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a1baa8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a14102bb0 0x7f6a14113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a1baa8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a141135b0 0x7f6a1410cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a1baa8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a14113bc0 con 0x7f6a141135b0 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a1baa8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1410d450 con 0x7f6a14102bb0 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a19043700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a141135b0 0x7f6a1410cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a19043700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a141135b0 0x7f6a1410cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56390/0 (socket says 172.21.15.2:56390) 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.294+0000 7f6a19043700 1 -- 172.21.15.2:0/712099156 learned_addr learned my addr 172.21.15.2:0/712099156 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a19043700 1 -- 172.21.15.2:0/712099156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a14102bb0 msgr2=0x7f6a14113070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a19844700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a14102bb0 0x7f6a14113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a19043700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a14102bb0 0x7f6a14113070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a19043700 1 -- 172.21.15.2:0/712099156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a10005040 con 0x7f6a141135b0 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a19043700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a141135b0 0x7f6a1410cf10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a0400b790 tx=0x7f6a04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a0a7fc700 1 -- 172.21.15.2:0/712099156 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a04010870 con 0x7f6a141135b0 2024-02-16T22:14:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1410d730 con 0x7f6a141135b0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a0a7fc700 1 -- 172.21.15.2:0/712099156 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a0400f3e0 con 0x7f6a141135b0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a0a7fc700 1 -- 172.21.15.2:0/712099156 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a04010ea0 con 0x7f6a141135b0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.295+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1410dcf0 con 0x7f6a141135b0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.296+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a14101ed0 con 0x7f6a141135b0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.296+0000 7f6a0a7fc700 1 -- 172.21.15.2:0/712099156 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f6a0400f550 con 0x7f6a141135b0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.297+0000 7f6a0a7fc700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6a00069380 0x7f6a0006b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.297+0000 7f6a0a7fc700 1 -- 172.21.15.2:0/712099156 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6a04086690 con 0x7f6a141135b0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.297+0000 7f6a19844700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6a00069380 0x7f6a0006b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.297+0000 7f6a19844700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6a00069380 0x7f6a0006b830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a10005010 tx=0x7f6a10009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.301+0000 7f6a0a7fc700 1 -- 172.21.15.2:0/712099156 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6a040574b0 con 0x7f6a141135b0 2024-02-16T22:14:43.726 INFO:teuthology.orchestra.run.smithi002.stdout:# minimal ceph.conf for 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 43c11c36-cd18-11ee-95ba-87774f69a715 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stdout: mon_host = [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.493+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6a1405f5c0 con 0x7f6a141135b0 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.493+0000 7f6a0a7fc700 1 -- 172.21.15.2:0/712099156 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7f6a040140f0 con 0x7f6a141135b0 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.495+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6a00069380 msgr2=0x7f6a0006b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.495+0000 7f6a1baa8700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6a00069380 0x7f6a0006b830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a10005010 tx=0x7f6a10009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.495+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a141135b0 msgr2=0x7f6a1410cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.495+0000 7f6a1baa8700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a141135b0 0x7f6a1410cf10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a0400b790 tx=0x7f6a04009250 comp rx=0 tx=0).stop 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.495+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 shutdown_connections 2024-02-16T22:14:43.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.495+0000 7f6a1baa8700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6a00069380 0x7f6a0006b830 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:43.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.495+0000 7f6a1baa8700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a14102bb0 0x7f6a14113070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:43.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.495+0000 7f6a1baa8700 1 --2- 172.21.15.2:0/712099156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a141135b0 0x7f6a1410cf10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:43.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.495+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 >> 172.21.15.2:0/712099156 conn(0x7f6a140f8760 msgr2=0x7f6a140fa6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:43.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.496+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 shutdown_connections 2024-02-16T22:14:43.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:43.496+0000 7f6a1baa8700 1 -- 172.21.15.2:0/712099156 wait complete. 2024-02-16T22:14:44.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:43 smithi002 ceph-mon[29154]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:43 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/712099156' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi002.resmjv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-16T22:14:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-16T22:14:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:44.268 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-16T22:14:44.268 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:14:44.268 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.conf 2024-02-16T22:14:44.299 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:14:44.299 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-16T22:14:44.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:43 smithi103 ceph-mon[36259]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:44.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:43 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/712099156' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:44.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:44.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:44.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi002.resmjv", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-16T22:14:44.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-16T22:14:44.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:44.366 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:14:44.366 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-02-16T22:14:44.401 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:14:44.401 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-16T22:14:44.471 INFO:tasks.cephadm:Deploying OSDs... 2024-02-16T22:14:44.471 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:14:44.471 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-02-16T22:14:44.486 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-16T22:14:44.486 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2024-02-16T22:14:44.541 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-16T22:14:44.542 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-16T22:14:44.542 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-16T22:14:44.542 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:14:44.542 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-02-16T22:14:44.542 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-02-16 22:13:56.894240878 +0000 2024-02-16T22:14:44.542 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-02-16 22:08:05.440264396 +0000 2024-02-16T22:14:44.542 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-02-16 22:08:05.440264396 +0000 2024-02-16T22:14:44.542 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-02-16 22:08:05.440264396 +0000 2024-02-16T22:14:44.542 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-16T22:14:44.607 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-02-16T22:14:44.607 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-02-16T22:14:44.607 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 9.9558e-05 s, 5.1 MB/s 2024-02-16T22:14:44.609 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-16T22:14:44.665 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2024-02-16T22:14:44.721 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-16T22:14:44.721 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-16T22:14:44.721 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-16T22:14:44.721 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:14:44.721 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-02-16T22:14:44.721 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-02-16 22:13:56.894240878 +0000 2024-02-16T22:14:44.721 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-02-16 22:08:05.775257231 +0000 2024-02-16T22:14:44.721 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-02-16 22:08:05.775257231 +0000 2024-02-16T22:14:44.721 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-02-16 22:08:05.775257231 +0000 2024-02-16T22:14:44.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-16T22:14:44.787 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-02-16T22:14:44.787 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-02-16T22:14:44.788 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000102319 s, 5.0 MB/s 2024-02-16T22:14:44.789 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-16T22:14:44.846 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2024-02-16T22:14:44.901 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-16T22:14:44.901 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-16T22:14:44.901 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-16T22:14:44.901 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:14:44.901 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-02-16T22:14:44.901 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-02-16 22:13:56.895240857 +0000 2024-02-16T22:14:44.901 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-02-16 22:08:06.123249789 +0000 2024-02-16T22:14:44.901 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-02-16 22:08:06.123249789 +0000 2024-02-16T22:14:44.901 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-02-16 22:08:06.123249789 +0000 2024-02-16T22:14:44.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-16T22:14:44.967 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-02-16T22:14:44.968 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-02-16T22:14:44.968 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000105806 s, 4.8 MB/s 2024-02-16T22:14:44.969 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-16T22:14:45.025 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2024-02-16T22:14:45.079 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-16T22:14:45.080 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-16T22:14:45.080 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-16T22:14:45.080 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:14:45.080 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-02-16T22:14:45.080 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-02-16 22:13:56.895240857 +0000 2024-02-16T22:14:45.080 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-02-16 22:08:06.449242817 +0000 2024-02-16T22:14:45.080 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-02-16 22:08:06.449242817 +0000 2024-02-16T22:14:45.080 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-02-16 22:08:06.449242817 +0000 2024-02-16T22:14:45.080 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-16T22:14:45.146 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:44 smithi002 ceph-mon[29154]: Reconfiguring mgr.smithi002.resmjv (unknown last config time)... 2024-02-16T22:14:45.146 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:44 smithi002 ceph-mon[29154]: Reconfiguring daemon mgr.smithi002.resmjv on smithi002 2024-02-16T22:14:45.148 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-02-16T22:14:45.148 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-02-16T22:14:45.148 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 8.4408e-05 s, 6.1 MB/s 2024-02-16T22:14:45.149 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-16T22:14:45.204 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:14:45.204 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-02-16T22:14:45.223 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-16T22:14:45.223 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-02-16T22:14:45.280 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-16T22:14:45.280 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-16T22:14:45.281 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-16T22:14:45.281 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:14:45.281 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-02-16T22:14:45.281 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-16 22:14:31.570889808 +0000 2024-02-16T22:14:45.281 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-16 22:08:04.033750628 +0000 2024-02-16T22:14:45.281 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-16 22:08:04.033750628 +0000 2024-02-16T22:14:45.281 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-16 22:08:04.033750628 +0000 2024-02-16T22:14:45.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-16T22:14:45.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:44 smithi103 ceph-mon[36259]: Reconfiguring mgr.smithi002.resmjv (unknown last config time)... 2024-02-16T22:14:45.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:44 smithi103 ceph-mon[36259]: Reconfiguring daemon mgr.smithi002.resmjv on smithi002 2024-02-16T22:14:45.349 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-16T22:14:45.349 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-16T22:14:45.349 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000357353 s, 1.4 MB/s 2024-02-16T22:14:45.350 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-16T22:14:45.408 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-02-16T22:14:45.465 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-16T22:14:45.465 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-16T22:14:45.465 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-16T22:14:45.465 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:14:45.465 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-02-16T22:14:45.466 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-16 22:14:31.570889808 +0000 2024-02-16T22:14:45.466 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-16 22:08:04.378743885 +0000 2024-02-16T22:14:45.466 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-16 22:08:04.378743885 +0000 2024-02-16T22:14:45.466 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-16 22:08:04.378743885 +0000 2024-02-16T22:14:45.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-16T22:14:45.533 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-16T22:14:45.533 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-16T22:14:45.533 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000257505 s, 2.0 MB/s 2024-02-16T22:14:45.534 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-16T22:14:45.591 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-02-16T22:14:45.647 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-16T22:14:45.647 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-16T22:14:45.647 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-16T22:14:45.647 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:14:45.647 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-02-16T22:14:45.647 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-16 22:14:31.571889789 +0000 2024-02-16T22:14:45.647 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-16 22:08:04.702737554 +0000 2024-02-16T22:14:45.647 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-16 22:08:04.702737554 +0000 2024-02-16T22:14:45.648 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-16 22:08:04.702737554 +0000 2024-02-16T22:14:45.648 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-16T22:14:45.714 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-16T22:14:45.715 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-16T22:14:45.715 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000231208 s, 2.2 MB/s 2024-02-16T22:14:45.716 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-16T22:14:45.773 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-02-16T22:14:45.829 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-16T22:14:45.829 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-16T22:14:45.829 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-16T22:14:45.829 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:14:45.829 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-02-16T22:14:45.829 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-02-16 22:14:31.571889789 +0000 2024-02-16T22:14:45.830 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-16 22:08:05.044730870 +0000 2024-02-16T22:14:45.830 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-16 22:08:05.044730870 +0000 2024-02-16T22:14:45.830 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-16 22:08:05.044730870 +0000 2024-02-16T22:14:45.830 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-16T22:14:45.896 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-02-16T22:14:45.896 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-02-16T22:14:45.896 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000221536 s, 2.3 MB/s 2024-02-16T22:14:45.898 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-16T22:14:45.953 INFO:tasks.cephadm:Deploying osd.0 on smithi002 with /dev/vg_nvme/lv_4... 2024-02-16T22:14:45.953 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-16T22:14:46.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:45 smithi103 ceph-mon[36259]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:46.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:46.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:46.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi002", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-16T22:14:46.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:46.155 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:45 smithi002 ceph-mon[29154]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:46.155 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:46.155 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:46.155 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi002", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-16T22:14:46.155 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:14:47.008 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:46 smithi002 ceph-mon[29154]: Reconfiguring crash.smithi002 (monmap changed)... 2024-02-16T22:14:47.008 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:46 smithi002 ceph-mon[29154]: Reconfiguring daemon crash.smithi002 on smithi002 2024-02-16T22:14:47.008 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:46 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:47.008 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:46 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:47.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:46 smithi103 ceph-mon[36259]: Reconfiguring crash.smithi002 (monmap changed)... 2024-02-16T22:14:47.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:46 smithi103 ceph-mon[36259]: Reconfiguring daemon crash.smithi002 on smithi002 2024-02-16T22:14:47.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:46 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:47.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:46 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:47.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:47 smithi002 ceph-mon[29154]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:47.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:47 smithi002 ceph-mon[29154]: Reconfiguring alertmanager.smithi002 (dependencies changed)... 2024-02-16T22:14:47.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:47 smithi002 ceph-mon[29154]: Reconfiguring daemon alertmanager.smithi002 on smithi002 2024-02-16T22:14:48.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:47 smithi103 ceph-mon[36259]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:48.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:47 smithi103 ceph-mon[36259]: Reconfiguring alertmanager.smithi002 (dependencies changed)... 2024-02-16T22:14:48.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:47 smithi103 ceph-mon[36259]: Reconfiguring daemon alertmanager.smithi002 on smithi002 2024-02-16T22:14:49.969 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:49 smithi002 ceph-mon[29154]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:49 smithi103 ceph-mon[36259]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-16T22:14:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-16T22:14:50.891 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-02-16T22:14:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-02-16T22:14:50.891 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00832402 s, 1.3 GB/s 2024-02-16T22:14:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-02-16T22:14:52.020 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:51 smithi002 ceph-mon[29154]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:52.021 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_4 2024-02-16T22:14:52.302 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:14:52.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:51 smithi103 ceph-mon[36259]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:54.150 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:53 smithi002 ceph-mon[29154]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:54.150 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:54.150 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:54.150 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:14:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:53 smithi103 ceph-mon[36259]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:14:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:14:54.924 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:54 smithi002 ceph-mon[29154]: Reconfiguring grafana.smithi002 (dependencies changed)... 2024-02-16T22:14:54.924 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:54 smithi002 ceph-mon[29154]: Reconfiguring daemon grafana.smithi002 on smithi002 2024-02-16T22:14:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.033+0000 7efff477d700 1 -- 172.21.15.2:0/1353500192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec0fda80 msgr2=0x7effec0ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.033+0000 7efff477d700 1 --2- 172.21.15.2:0/1353500192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec0fda80 0x7effec0ffe60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7effdc004940 tx=0x7effdc009e30 comp rx=0 tx=0).stop 2024-02-16T22:14:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.033+0000 7efff477d700 1 -- 172.21.15.2:0/1353500192 shutdown_connections 2024-02-16T22:14:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.033+0000 7efff477d700 1 --2- 172.21.15.2:0/1353500192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effec1003a0 0x7effec102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.033+0000 7efff477d700 1 --2- 172.21.15.2:0/1353500192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec0fda80 0x7effec0ffe60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.033+0000 7efff477d700 1 -- 172.21.15.2:0/1353500192 >> 172.21.15.2:0/1353500192 conn(0x7effec0f74e0 msgr2=0x7effec0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:14:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.033+0000 7efff477d700 1 -- 172.21.15.2:0/1353500192 shutdown_connections 2024-02-16T22:14:55.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.033+0000 7efff477d700 1 -- 172.21.15.2:0/1353500192 wait complete. 2024-02-16T22:14:55.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.034+0000 7efff477d700 1 Processor -- start 2024-02-16T22:14:55.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.034+0000 7efff477d700 1 -- start start 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.034+0000 7efff477d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effec0fda80 0x7effec10d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.034+0000 7efff477d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec1003a0 0x7effec10de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.034+0000 7efff477d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec10e470 con 0x7effec1003a0 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.034+0000 7efff477d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec10ed10 con 0x7effec0fda80 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.034+0000 7efff1d18700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec1003a0 0x7effec10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.034+0000 7efff1d18700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec1003a0 0x7effec10de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48184/0 (socket says 172.21.15.2:48184) 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.034+0000 7efff1d18700 1 -- 172.21.15.2:0/238684461 learned_addr learned my addr 172.21.15.2:0/238684461 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7efff2519700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effec0fda80 0x7effec10d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7efff1d18700 1 -- 172.21.15.2:0/238684461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effec0fda80 msgr2=0x7effec10d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7efff1d18700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effec0fda80 0x7effec10d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7efff1d18700 1 -- 172.21.15.2:0/238684461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc005040 con 0x7effec1003a0 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7efff1d18700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec1003a0 0x7effec10de70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7effe800c8c0 tx=0x7effe80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7effe2ffd700 1 -- 172.21.15.2:0/238684461 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe800a980 con 0x7effec1003a0 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7effe2ffd700 1 -- 172.21.15.2:0/238684461 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effe800ce60 con 0x7effec1003a0 2024-02-16T22:14:55.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effec10eff0 con 0x7effec1003a0 2024-02-16T22:14:55.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7effe2ffd700 1 -- 172.21.15.2:0/238684461 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe80115e0 con 0x7effec1003a0 2024-02-16T22:14:55.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.035+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effec10f530 con 0x7effec1003a0 2024-02-16T22:14:55.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.036+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effec04bed0 con 0x7effec1003a0 2024-02-16T22:14:55.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.037+0000 7effe2ffd700 1 -- 172.21.15.2:0/238684461 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7effe8012400 con 0x7effec1003a0 2024-02-16T22:14:55.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.037+0000 7effe2ffd700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7effd8069210 0x7effd806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:14:55.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.037+0000 7effe2ffd700 1 -- 172.21.15.2:0/238684461 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7effe8086080 con 0x7effec1003a0 2024-02-16T22:14:55.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.037+0000 7efff2519700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7effd8069210 0x7effd806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:14:55.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.038+0000 7efff2519700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7effd8069210 0x7effd806b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7effdc004f90 tx=0x7effdc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:14:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.042+0000 7effe2ffd700 1 -- 172.21.15.2:0/238684461 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7effe8056fc0 con 0x7effec1003a0 2024-02-16T22:14:55.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:14:55.248+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7effec10f990 con 0x7effd8069210 2024-02-16T22:14:55.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:54 smithi103 ceph-mon[36259]: Reconfiguring grafana.smithi002 (dependencies changed)... 2024-02-16T22:14:55.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:54 smithi103 ceph-mon[36259]: Reconfiguring daemon grafana.smithi002 on smithi002 2024-02-16T22:14:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:55 smithi002 ceph-mon[29154]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:14:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:14:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:56.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:55 smithi103 ceph-mon[36259]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:56.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:14:56.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:14:56.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:14:57.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:56 smithi002 ceph-mon[29154]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:14:57.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:56 smithi103 ceph-mon[36259]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:14:58.052 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:57 smithi002 ceph-mon[29154]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:58.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:57 smithi103 ceph-mon[36259]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:58.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:14:58 smithi002 ceph-mon[29154]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:14:59.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:14:58 smithi103 ceph-mon[36259]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:00.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:00 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:00.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:00 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:00.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:00 smithi002 ceph-mon[29154]: Reconfiguring prometheus.smithi002 (dependencies changed)... 2024-02-16T22:15:00.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:00 smithi002 ceph-mon[29154]: Reconfiguring daemon prometheus.smithi002 on smithi002 2024-02-16T22:15:00.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:00 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:00.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:00 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:00.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:00 smithi103 ceph-mon[36259]: Reconfiguring prometheus.smithi002 (dependencies changed)... 2024-02-16T22:15:00.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:00 smithi103 ceph-mon[36259]: Reconfiguring daemon prometheus.smithi002 on smithi002 2024-02-16T22:15:01.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:01 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3517142308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "348575cf-dd0d-444c-bab9-bd51dcc9189c"}]: dispatch 2024-02-16T22:15:01.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:01 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3517142308' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "348575cf-dd0d-444c-bab9-bd51dcc9189c"}]': finished 2024-02-16T22:15:01.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:01 smithi103 ceph-mon[36259]: osdmap e6: 1 total, 0 up, 1 in 2024-02-16T22:15:01.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:01 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:01.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:01 smithi103 ceph-mon[36259]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:01.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:01 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3110846278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:15:01.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:01 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3517142308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "348575cf-dd0d-444c-bab9-bd51dcc9189c"}]: dispatch 2024-02-16T22:15:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:01 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3517142308' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "348575cf-dd0d-444c-bab9-bd51dcc9189c"}]': finished 2024-02-16T22:15:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:01 smithi002 ceph-mon[29154]: osdmap e6: 1 total, 0 up, 1 in 2024-02-16T22:15:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:01 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:01 smithi002 ceph-mon[29154]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:01.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:01 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3110846278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:15:03.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:03 smithi002 ceph-mon[29154]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:04.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:03 smithi103 ceph-mon[36259]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:05.967 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:05 smithi002 ceph-mon[29154]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:06.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:05 smithi103 ceph-mon[36259]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:07.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:07 smithi002 ceph-mon[29154]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:08.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:07 smithi103 ceph-mon[36259]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:08.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:15:09.226 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:08 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:15:09.931 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:09 smithi002 ceph-mon[29154]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:09.931 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:09.931 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:09.931 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-16T22:15:09.932 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:10.094 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:09 smithi103 ceph-mon[36259]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:10.095 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:09 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:10.095 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:09 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:10.095 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:09 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi103", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-16T22:15:10.095 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:09 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:11.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:10 smithi103 ceph-mon[36259]: Reconfiguring crash.smithi103 (monmap changed)... 2024-02-16T22:15:11.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:10 smithi103 ceph-mon[36259]: Reconfiguring daemon crash.smithi103 on smithi103 2024-02-16T22:15:11.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:10 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-16T22:15:11.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:10 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:11.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:10 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:11.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:10 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:11.193 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:10 smithi002 ceph-mon[29154]: Reconfiguring crash.smithi103 (monmap changed)... 2024-02-16T22:15:11.194 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:10 smithi002 ceph-mon[29154]: Reconfiguring daemon crash.smithi103 on smithi103 2024-02-16T22:15:11.194 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:10 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-16T22:15:11.194 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:10 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:11.194 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:10 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:11.194 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:10 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:11.976 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:11 smithi103 ceph-mon[36259]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:11.976 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:11 smithi103 ceph-mon[36259]: Deploying daemon osd.0 on smithi002 2024-02-16T22:15:11.976 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:11 smithi103 ceph-mon[36259]: Reconfiguring mgr.smithi103.ydydbo (monmap changed)... 2024-02-16T22:15:11.976 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.ydydbo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-16T22:15:11.976 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-16T22:15:11.976 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:11.976 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:11 smithi103 ceph-mon[36259]: Reconfiguring daemon mgr.smithi103.ydydbo on smithi103 2024-02-16T22:15:12.181 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:11 smithi002 ceph-mon[29154]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:12.181 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:11 smithi002 ceph-mon[29154]: Deploying daemon osd.0 on smithi002 2024-02-16T22:15:12.181 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:11 smithi002 ceph-mon[29154]: Reconfiguring mgr.smithi103.ydydbo (monmap changed)... 2024-02-16T22:15:12.182 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi103.ydydbo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-16T22:15:12.182 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-16T22:15:12.182 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:12.182 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:11 smithi002 ceph-mon[29154]: Reconfiguring daemon mgr.smithi103.ydydbo on smithi103 2024-02-16T22:15:13.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:13.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:13.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:13 smithi002 ceph-mon[29154]: Reconfiguring mon.smithi103 (monmap changed)... 2024-02-16T22:15:13.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-16T22:15:13.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-16T22:15:13.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:13 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:13.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:13 smithi002 ceph-mon[29154]: Reconfiguring daemon mon.smithi103 on smithi103 2024-02-16T22:15:13.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:13 smithi002 ceph-mon[29154]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:13.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:13 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:13.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:13 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:13.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:13 smithi103 ceph-mon[36259]: Reconfiguring mon.smithi103 (monmap changed)... 2024-02-16T22:15:13.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:13 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-16T22:15:13.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:13 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-16T22:15:13.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:13 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:13.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:13 smithi103 ceph-mon[36259]: Reconfiguring daemon mon.smithi103 on smithi103 2024-02-16T22:15:13.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:13 smithi103 ceph-mon[36259]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:14.956 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi002.front.sepia.ceph.com:3000"}]: dispatch 2024-02-16T22:15:14.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi002.front.sepia.ceph.com:3000"}]: dispatch 2024-02-16T22:15:14.958 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:14.958 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-16T22:15:14.958 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-16T22:15:14.958 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi002.front.sepia.ceph.com:9095"}]: dispatch 2024-02-16T22:15:14.958 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi002.front.sepia.ceph.com:9095"}]: dispatch 2024-02-16T22:15:14.958 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:14.958 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:15:15.054 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi002.front.sepia.ceph.com:3000"}]: dispatch 2024-02-16T22:15:15.055 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi002.front.sepia.ceph.com:3000"}]: dispatch 2024-02-16T22:15:15.056 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:15.056 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-16T22:15:15.056 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-16T22:15:15.056 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi002.front.sepia.ceph.com:9095"}]: dispatch 2024-02-16T22:15:15.056 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi002.front.sepia.ceph.com:9095"}]: dispatch 2024-02-16T22:15:15.056 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:15.056 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:15:16.091 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:15 smithi002 ceph-mon[29154]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:16.091 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:16.091 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:15 smithi103 ceph-mon[36259]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:16.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:16.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:17.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:17 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:17.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:17 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:17.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:17 smithi002 ceph-mon[29154]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:17.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:17 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:17.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:17 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:17.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:17 smithi103 ceph-mon[36259]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:19.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:19 smithi002 ceph-mon[29154]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:19.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.870+0000 7effe2ffd700 1 -- 172.21.15.2:0/238684461 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7effec10f990 con 0x7effd8069210 2024-02-16T22:15:19.872 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 0 on host 'smithi002' 2024-02-16T22:15:19.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7effd8069210 msgr2=0x7effd806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7effd8069210 0x7effd806b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7effdc004f90 tx=0x7effdc009e30 comp rx=0 tx=0).stop 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec1003a0 msgr2=0x7effec10de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec1003a0 0x7effec10de70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7effe800c8c0 tx=0x7effe80044c0 comp rx=0 tx=0).stop 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 shutdown_connections 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7effd8069210 0x7effd806b6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effec0fda80 0x7effec10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 --2- 172.21.15.2:0/238684461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effec1003a0 0x7effec10de70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 >> 172.21.15.2:0/238684461 conn(0x7effec0f74e0 msgr2=0x7effec0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 shutdown_connections 2024-02-16T22:15:19.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:19.872+0000 7efff477d700 1 -- 172.21.15.2:0/238684461 wait complete. 2024-02-16T22:15:19.938 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:19 smithi103 ceph-mon[36259]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:20.819 DEBUG:teuthology.orchestra.run.smithi002:osd.0> sudo journalctl -f -n 0 -u ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.0.service 2024-02-16T22:15:20.822 INFO:tasks.cephadm:Deploying osd.1 on smithi002 with /dev/vg_nvme/lv_3... 2024-02-16T22:15:20.822 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-16T22:15:21.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:20 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:21.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:20 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:21.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:20 smithi103 ceph-mon[36259]: from='osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-16T22:15:21.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:20 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:21.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:20 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:21.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:20 smithi002 ceph-mon[29154]: from='osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-16T22:15:21.237 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:15:21.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:21 smithi002 ceph-mon[29154]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:21.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:21 smithi002 ceph-mon[29154]: from='osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-16T22:15:21.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:21 smithi002 ceph-mon[29154]: osdmap e7: 1 total, 0 up, 1 in 2024-02-16T22:15:21.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:21 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:21.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:21 smithi002 ceph-mon[29154]: from='osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-16T22:15:21.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:15:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:15:21.848+0000 7f2daa1d9700 -1 osd.0 0 waiting for initial osdmap 2024-02-16T22:15:21.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:15:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:15:21.852+0000 7f2da4fd4700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-16T22:15:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:21 smithi103 ceph-mon[36259]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:21 smithi103 ceph-mon[36259]: from='osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-16T22:15:22.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:21 smithi103 ceph-mon[36259]: osdmap e7: 1 total, 0 up, 1 in 2024-02-16T22:15:22.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:21 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:22.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:21 smithi103 ceph-mon[36259]: from='osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-16T22:15:22.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:22 smithi002 ceph-mon[29154]: from='osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-16T22:15:22.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:22 smithi002 ceph-mon[29154]: osdmap e8: 1 total, 0 up, 1 in 2024-02-16T22:15:22.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:22.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:22.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:22.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:23.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:22 smithi103 ceph-mon[36259]: from='osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-16T22:15:23.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:22 smithi103 ceph-mon[36259]: osdmap e8: 1 total, 0 up, 1 in 2024-02-16T22:15:23.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:23.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:23.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:23.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:23.836 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-16T22:15:23.836 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-16T22:15:23.836 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-02-16T22:15:23.836 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-02-16T22:15:23.836 INFO:teuthology.orchestra.run.smithi002.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127214 s, 824 MB/s 2024-02-16T22:15:23.836 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 2024-02-16T22:15:23.836 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-02-16T22:15:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:23 smithi002 ceph-mon[29154]: purged_snaps scrub starts 2024-02-16T22:15:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:23 smithi002 ceph-mon[29154]: purged_snaps scrub ok 2024-02-16T22:15:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:23 smithi002 ceph-mon[29154]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:23 smithi002 ceph-mon[29154]: osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] boot 2024-02-16T22:15:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:23 smithi002 ceph-mon[29154]: osdmap e9: 1 total, 1 up, 1 in 2024-02-16T22:15:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:15:24.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:23 smithi103 ceph-mon[36259]: purged_snaps scrub starts 2024-02-16T22:15:24.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:23 smithi103 ceph-mon[36259]: purged_snaps scrub ok 2024-02-16T22:15:24.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:23 smithi103 ceph-mon[36259]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-16T22:15:24.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:23 smithi103 ceph-mon[36259]: osd.0 [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] boot 2024-02-16T22:15:24.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:23 smithi103 ceph-mon[36259]: osdmap e9: 1 total, 1 up, 1 in 2024-02-16T22:15:24.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-16T22:15:24.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:15:25.160 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_3 2024-02-16T22:15:25.443 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:25 smithi002 ceph-mon[29154]: osdmap e10: 1 total, 1 up, 1 in 2024-02-16T22:15:25.443 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:25 smithi002 ceph-mon[29154]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:25.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:25 smithi103 ceph-mon[36259]: osdmap e10: 1 total, 1 up, 1 in 2024-02-16T22:15:25.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:25 smithi103 ceph-mon[36259]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:25.695 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:15:27.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 -- 172.21.15.2:0/2151637992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 msgr2=0x7f5814102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:15:27.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 --2- 172.21.15.2:0/2151637992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 0x7f5814102520 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f57fc004880 tx=0x7f57fc009e30 comp rx=0 tx=0).stop 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 -- 172.21.15.2:0/2151637992 shutdown_connections 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 --2- 172.21.15.2:0/2151637992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 0x7f5814102520 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 --2- 172.21.15.2:0/2151637992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5814066c70 0x7f58140ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 -- 172.21.15.2:0/2151637992 >> 172.21.15.2:0/2151637992 conn(0x7f58140f7520 msgr2=0x7f58140f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 -- 172.21.15.2:0/2151637992 shutdown_connections 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 -- 172.21.15.2:0/2151637992 wait complete. 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 Processor -- start 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.698+0000 7f581a134700 1 -- start start 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f581a134700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5814066c70 0x7f58140fd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f581a134700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 0x7f58140fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f581a134700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58140fbff0 con 0x7f58141000c0 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f581a134700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58140fc160 con 0x7f5814066c70 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f5812ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 0x7f58140fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:15:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f5812ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 0x7f58140fb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39352/0 (socket says 172.21.15.2:39352) 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f5812ffd700 1 -- 172.21.15.2:0/1318039167 learned_addr learned my addr 172.21.15.2:0/1318039167 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f58137fe700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5814066c70 0x7f58140fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f5812ffd700 1 -- 172.21.15.2:0/1318039167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5814066c70 msgr2=0x7f58140fd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f5812ffd700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5814066c70 0x7f58140fd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f5812ffd700 1 -- 172.21.15.2:0/1318039167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc005040 con 0x7f58141000c0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f5812ffd700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 0x7f58140fb9f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57fc008d00 tx=0x7f57fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f580bfff700 1 -- 172.21.15.2:0/1318039167 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57fc00e030 con 0x7f58141000c0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f580bfff700 1 -- 172.21.15.2:0/1318039167 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57fc013e60 con 0x7f58141000c0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58140fc3e0 con 0x7f58141000c0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.699+0000 7f580bfff700 1 -- 172.21.15.2:0/1318039167 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57fc01ce40 con 0x7f58141000c0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.700+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5814115960 con 0x7f58141000c0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.701+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5814107a50 con 0x7f58141000c0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.701+0000 7f580bfff700 1 -- 172.21.15.2:0/1318039167 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f57fc01c690 con 0x7f58141000c0 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.701+0000 7f580bfff700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5800069210 0x7f580006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:15:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.701+0000 7f580bfff700 1 -- 172.21.15.2:0/1318039167 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f57fc087c40 con 0x7f58141000c0 2024-02-16T22:15:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.702+0000 7f58137fe700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5800069210 0x7f580006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:15:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.702+0000 7f58137fe700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5800069210 0x7f580006b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f58140fd300 tx=0x7f580400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:15:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.706+0000 7f580bfff700 1 -- 172.21.15.2:0/1318039167 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f57fc058920 con 0x7f58141000c0 2024-02-16T22:15:27.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:27.922+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f581405e3d0 con 0x7f5800069210 2024-02-16T22:15:27.924 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:27 smithi002 ceph-mon[29154]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:27.924 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:27.924 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:27.924 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:15:27.924 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:27.925 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:27.925 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:15:27.925 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:27.925 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:15:28.050 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:27 smithi103 ceph-mon[36259]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:28.050 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:28.050 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:28.050 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:15:28.050 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:28.050 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:28.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:15:28.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:28.051 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:15:28.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:28 smithi002 ceph-mon[29154]: Detected new or changed devices on smithi002 2024-02-16T22:15:28.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:28 smithi002 ceph-mon[29154]: Adjusting osd_memory_target on smithi002 to 12869M 2024-02-16T22:15:28.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:28 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:15:28.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:28 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:15:28.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:28 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:28.962 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:28 smithi103 ceph-mon[36259]: Detected new or changed devices on smithi002 2024-02-16T22:15:28.962 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:28 smithi103 ceph-mon[36259]: Adjusting osd_memory_target on smithi002 to 12869M 2024-02-16T22:15:28.962 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:28 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:15:28.962 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:28 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:15:28.962 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:28 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:29.893 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:29 smithi002 ceph-mon[29154]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:15:29.893 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:29 smithi002 ceph-mon[29154]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:29.893 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:29 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:29.893 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:29 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:30.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:29 smithi103 ceph-mon[36259]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:15:30.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:29 smithi103 ceph-mon[36259]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:30.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:29 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:30.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:29 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:31.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:31 smithi002 ceph-mon[29154]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:31.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:31 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3871613804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eaffda16-c0db-440f-9a1a-99e228b3dc19"}]: dispatch 2024-02-16T22:15:31.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:31 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3871613804' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eaffda16-c0db-440f-9a1a-99e228b3dc19"}]': finished 2024-02-16T22:15:31.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:31 smithi002 ceph-mon[29154]: osdmap e11: 2 total, 1 up, 2 in 2024-02-16T22:15:31.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:31 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:32.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:31 smithi103 ceph-mon[36259]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:32.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:31 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3871613804' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eaffda16-c0db-440f-9a1a-99e228b3dc19"}]: dispatch 2024-02-16T22:15:32.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:31 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3871613804' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eaffda16-c0db-440f-9a1a-99e228b3dc19"}]': finished 2024-02-16T22:15:32.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:31 smithi103 ceph-mon[36259]: osdmap e11: 2 total, 1 up, 2 in 2024-02-16T22:15:32.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:31 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:33.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:32 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1979512496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:15:33.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:32 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:33.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:32 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1979512496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:15:33.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:32 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:33.853 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:33 smithi103 ceph-mon[36259]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:33.853 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:33.853 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:33.853 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:15:33.854 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:34.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:33 smithi002 ceph-mon[29154]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:15:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:36.108 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:35 smithi002 ceph-mon[29154]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:36.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:35 smithi103 ceph-mon[36259]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:37.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:37 smithi002 ceph-mon[29154]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:38.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:37 smithi103 ceph-mon[36259]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:38.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-16T22:15:38.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:38.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:15:39.227 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-16T22:15:39.227 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:39.227 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:15:40.130 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:39 smithi002 ceph-mon[29154]: Deploying daemon osd.1 on smithi002 2024-02-16T22:15:40.130 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:39 smithi002 ceph-mon[29154]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:40.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:39 smithi103 ceph-mon[36259]: Deploying daemon osd.1 on smithi002 2024-02-16T22:15:40.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:39 smithi103 ceph-mon[36259]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:42.191 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:41 smithi002 ceph-mon[29154]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:42.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:41 smithi103 ceph-mon[36259]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:44.023 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:43 smithi002 ceph-mon[29154]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:44.023 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:44.023 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:44.023 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:15:44.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:43 smithi103 ceph-mon[36259]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:44.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:44.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:44.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:15:46.139 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:45 smithi002 ceph-mon[29154]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:46.139 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:46.139 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:46.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:45 smithi103 ceph-mon[36259]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:46.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:46.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:46.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.726+0000 7f580bfff700 1 -- 172.21.15.2:0/1318039167 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f581405e3d0 con 0x7f5800069210 2024-02-16T22:15:46.729 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 1 on host 'smithi002' 2024-02-16T22:15:46.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5800069210 msgr2=0x7f580006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:15:46.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5800069210 0x7f580006b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f58140fd300 tx=0x7f580400b040 comp rx=0 tx=0).stop 2024-02-16T22:15:46.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 msgr2=0x7f58140fb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:15:46.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 0x7f58140fb9f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57fc008d00 tx=0x7f57fc009e30 comp rx=0 tx=0).stop 2024-02-16T22:15:46.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 shutdown_connections 2024-02-16T22:15:46.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5800069210 0x7f580006b6c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:46.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5814066c70 0x7f58140fd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:46.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 --2- 172.21.15.2:0/1318039167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58141000c0 0x7f58140fb9f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:46.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 >> 172.21.15.2:0/1318039167 conn(0x7f58140f7520 msgr2=0x7f58140f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:15:46.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 shutdown_connections 2024-02-16T22:15:46.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:46.728+0000 7f581a134700 1 -- 172.21.15.2:0/1318039167 wait complete. 2024-02-16T22:15:47.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:47 smithi002 ceph-mon[29154]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:47.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:47 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:47.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:47 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:47.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:47 smithi002 ceph-mon[29154]: from='osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-16T22:15:48.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:47 smithi103 ceph-mon[36259]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:48.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:47 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:48.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:47 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:48.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:47 smithi103 ceph-mon[36259]: from='osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-16T22:15:48.304 DEBUG:teuthology.orchestra.run.smithi002:osd.1> sudo journalctl -f -n 0 -u ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.1.service 2024-02-16T22:15:48.307 INFO:tasks.cephadm:Deploying osd.2 on smithi002 with /dev/vg_nvme/lv_2... 2024-02-16T22:15:48.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-16T22:15:49.055 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:15:49.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:48 smithi002 ceph-mon[29154]: from='osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-16T22:15:49.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:48 smithi002 ceph-mon[29154]: osdmap e12: 2 total, 1 up, 2 in 2024-02-16T22:15:49.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:48 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:49.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:48 smithi002 ceph-mon[29154]: from='osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-16T22:15:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:15:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:15:48.878+0000 7f637606f700 -1 osd.1 0 waiting for initial osdmap 2024-02-16T22:15:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:15:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:15:48.882+0000 7f636d663700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-16T22:15:49.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:48 smithi103 ceph-mon[36259]: from='osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-16T22:15:49.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:48 smithi103 ceph-mon[36259]: osdmap e12: 2 total, 1 up, 2 in 2024-02-16T22:15:49.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:48 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:49.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:48 smithi103 ceph-mon[36259]: from='osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-16T22:15:50.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:49 smithi002 ceph-mon[29154]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:50.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:49 smithi002 ceph-mon[29154]: from='osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-16T22:15:50.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:49 smithi002 ceph-mon[29154]: osdmap e13: 2 total, 1 up, 2 in 2024-02-16T22:15:50.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:49 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:50.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:49 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:49 smithi103 ceph-mon[36259]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-16T22:15:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:49 smithi103 ceph-mon[36259]: from='osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-16T22:15:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:49 smithi103 ceph-mon[36259]: osdmap e13: 2 total, 1 up, 2 in 2024-02-16T22:15:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:49 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:49 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:51.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:50 smithi002 ceph-mon[29154]: purged_snaps scrub starts 2024-02-16T22:15:51.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:50 smithi002 ceph-mon[29154]: purged_snaps scrub ok 2024-02-16T22:15:51.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:50 smithi002 ceph-mon[29154]: osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] boot 2024-02-16T22:15:51.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:50 smithi002 ceph-mon[29154]: osdmap e14: 2 total, 2 up, 2 in 2024-02-16T22:15:51.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:50 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:51.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:50 smithi002 ceph-mon[29154]: pgmap v53: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:15:51.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:50 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:51.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:50 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:51.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:50 smithi103 ceph-mon[36259]: purged_snaps scrub starts 2024-02-16T22:15:51.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:50 smithi103 ceph-mon[36259]: purged_snaps scrub ok 2024-02-16T22:15:51.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:50 smithi103 ceph-mon[36259]: osd.1 [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] boot 2024-02-16T22:15:51.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:50 smithi103 ceph-mon[36259]: osdmap e14: 2 total, 2 up, 2 in 2024-02-16T22:15:51.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:50 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-16T22:15:51.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:50 smithi103 ceph-mon[36259]: pgmap v53: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:15:51.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:50 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:51.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:50 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:51.482 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-16T22:15:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-16T22:15:51.483 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-02-16T22:15:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-02-16T22:15:51.483 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123455 s, 849 MB/s 2024-02-16T22:15:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-02-16T22:15:52.160 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:51 smithi002 ceph-mon[29154]: osdmap e15: 2 total, 2 up, 2 in 2024-02-16T22:15:52.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:51 smithi103 ceph-mon[36259]: osdmap e15: 2 total, 2 up, 2 in 2024-02-16T22:15:52.863 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_2 2024-02-16T22:15:53.161 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:52 smithi002 ceph-mon[29154]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:15:53.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:52 smithi103 ceph-mon[36259]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:15:53.368 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:15:54.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:15:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:15:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.518+0000 7f9ad1d20700 1 -- 172.21.15.2:0/1845645773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fcef0 msgr2=0x7f9acc0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:15:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.519+0000 7f9ad1d20700 1 --2- 172.21.15.2:0/1845645773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fcef0 0x7f9acc0fd320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9abc004940 tx=0x7f9abc009e30 comp rx=0 tx=0).stop 2024-02-16T22:15:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.519+0000 7f9ad1d20700 1 -- 172.21.15.2:0/1845645773 shutdown_connections 2024-02-16T22:15:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.519+0000 7f9ad1d20700 1 --2- 172.21.15.2:0/1845645773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9acc0fe0e0 0x7f9acc0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.519+0000 7f9ad1d20700 1 --2- 172.21.15.2:0/1845645773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fcef0 0x7f9acc0fd320 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.519+0000 7f9ad1d20700 1 -- 172.21.15.2:0/1845645773 >> 172.21.15.2:0/1845645773 conn(0x7f9acc0f8620 msgr2=0x7f9acc0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:15:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.519+0000 7f9ad1d20700 1 -- 172.21.15.2:0/1845645773 shutdown_connections 2024-02-16T22:15:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.519+0000 7f9ad1d20700 1 -- 172.21.15.2:0/1845645773 wait complete. 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.519+0000 7f9ad1d20700 1 Processor -- start 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.519+0000 7f9ad1d20700 1 -- start start 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9ad1d20700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9acc0fcef0 0x7f9acc071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9ad1d20700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fe0e0 0x7f9acc0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9ad1d20700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc0708e0 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9ad1d20700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc070a50 con 0x7f9acc0fcef0 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9acaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fe0e0 0x7f9acc0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9acaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fe0e0 0x7f9acc0702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34826/0 (socket says 172.21.15.2:34826) 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9acaffd700 1 -- 172.21.15.2:0/3123204697 learned_addr learned my addr 172.21.15.2:0/3123204697 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9acb7fe700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9acc0fcef0 0x7f9acc071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9acaffd700 1 -- 172.21.15.2:0/3123204697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9acc0fcef0 msgr2=0x7f9acc071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9acaffd700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9acc0fcef0 0x7f9acc071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9acaffd700 1 -- 172.21.15.2:0/3123204697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc005040 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9acaffd700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fe0e0 0x7f9acc0702e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0002a50 tx=0x7f9ac00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.520+0000 7f9ab3fff700 1 -- 172.21.15.2:0/3123204697 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ac0007cc0 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.521+0000 7f9ab3fff700 1 -- 172.21.15.2:0/3123204697 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ac000f040 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.521+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9acc070d30 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.521+0000 7f9ab3fff700 1 -- 172.21.15.2:0/3123204697 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ac000da30 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.521+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9acc1159a0 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.522+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9acc04bed0 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.522+0000 7f9ab3fff700 1 -- 172.21.15.2:0/3123204697 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f9ac000db90 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.523+0000 7f9ab3fff700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9ab4069210 0x7f9ab406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.523+0000 7f9ab3fff700 1 -- 172.21.15.2:0/3123204697 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f9ac0086270 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.523+0000 7f9acb7fe700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9ab4069210 0x7f9ab406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:15:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.523+0000 7f9acb7fe700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9ab4069210 0x7f9ab406b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00b840 tx=0x7f9abc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:15:55.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.527+0000 7f9ab3fff700 1 -- 172.21.15.2:0/3123204697 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9ac0056cf0 con 0x7f9acc0fe0e0 2024-02-16T22:15:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:15:55.745+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9acc102ac0 con 0x7f9ab4069210 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: Detected new or changed devices on smithi002 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: Adjusting osd_memory_target on smithi002 to 6434M 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:55.794 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:15:56.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: Detected new or changed devices on smithi002 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: Adjusting osd_memory_target on smithi002 to 6434M 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:15:56.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:15:56.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:56 smithi002 ceph-mon[29154]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:15:56.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:56 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:15:56.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:56 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:15:56.891 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:56 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:57.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:56 smithi103 ceph-mon[36259]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:15:57.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:56 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:15:57.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:56 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:15:57.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:56 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:15:57.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:57 smithi002 ceph-mon[29154]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:15:58.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:57 smithi103 ceph-mon[36259]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:15:59.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:15:59 smithi002 ceph-mon[29154]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:00.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:15:59 smithi103 ceph-mon[36259]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:00.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:00 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2332157986' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "732c0a64-5183-42aa-8220-a12116e52442"}]: dispatch 2024-02-16T22:16:00.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:00 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2332157986' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "732c0a64-5183-42aa-8220-a12116e52442"}]': finished 2024-02-16T22:16:00.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:00 smithi002 ceph-mon[29154]: osdmap e16: 3 total, 2 up, 3 in 2024-02-16T22:16:00.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:00 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:00.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:00 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/4131359210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:16:01.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:00 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2332157986' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "732c0a64-5183-42aa-8220-a12116e52442"}]: dispatch 2024-02-16T22:16:01.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:00 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2332157986' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "732c0a64-5183-42aa-8220-a12116e52442"}]': finished 2024-02-16T22:16:01.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:00 smithi103 ceph-mon[36259]: osdmap e16: 3 total, 2 up, 3 in 2024-02-16T22:16:01.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:00 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:01.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:00 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/4131359210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:16:02.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:01 smithi103 ceph-mon[36259]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:02.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:01 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:02.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:01 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:02.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:01 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:02.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:01 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:02.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:01 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:01 smithi002 ceph-mon[29154]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:01 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:01 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:01 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:02.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:01 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:02.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:01 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:03.897 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:03 smithi002 ceph-mon[29154]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:04.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:03 smithi103 ceph-mon[36259]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:06.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:05 smithi002 ceph-mon[29154]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:06.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:05 smithi103 ceph-mon[36259]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:08.209 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:07 smithi002 ceph-mon[29154]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:08.210 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-16T22:16:08.210 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:08.210 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:07 smithi002 ceph-mon[29154]: Deploying daemon osd.2 on smithi002 2024-02-16T22:16:08.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:07 smithi103 ceph-mon[36259]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:08.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-16T22:16:08.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:08.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:07 smithi103 ceph-mon[36259]: Deploying daemon osd.2 on smithi002 2024-02-16T22:16:09.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:16:09.226 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:08 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:16:10.185 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:09 smithi002 ceph-mon[29154]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:10.242 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:09 smithi103 ceph-mon[36259]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:12.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:11 smithi002 ceph-mon[29154]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:12.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:11 smithi103 ceph-mon[36259]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:13.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:12 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:13.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:12 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:13.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:12 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:16:13.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:12 smithi002 ceph-mon[29154]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:13.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:12 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:13.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:12 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:13.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:12 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:16:13.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:12 smithi103 ceph-mon[36259]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:15.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.406+0000 7f9ab3fff700 1 -- 172.21.15.2:0/3123204697 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9acc102ac0 con 0x7f9ab4069210 2024-02-16T22:16:15.408 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 2 on host 'smithi002' 2024-02-16T22:16:15.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9ab4069210 msgr2=0x7f9ab406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:16:15.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9ab4069210 0x7f9ab406b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00b840 tx=0x7f9abc009e30 comp rx=0 tx=0).stop 2024-02-16T22:16:15.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fe0e0 msgr2=0x7f9acc0702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:16:15.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fe0e0 0x7f9acc0702e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0002a50 tx=0x7f9ac00046b0 comp rx=0 tx=0).stop 2024-02-16T22:16:15.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 shutdown_connections 2024-02-16T22:16:15.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9ab4069210 0x7f9ab406b6c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:15.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9acc0fcef0 0x7f9acc071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:15.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 --2- 172.21.15.2:0/3123204697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9acc0fe0e0 0x7f9acc0702e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:15.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 >> 172.21.15.2:0/3123204697 conn(0x7f9acc0f8620 msgr2=0x7f9acc1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:16:15.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 shutdown_connections 2024-02-16T22:16:15.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:16:15.408+0000 7f9ad1d20700 1 -- 172.21.15.2:0/3123204697 wait complete. 2024-02-16T22:16:16.136 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:15 smithi002 ceph-mon[29154]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:16.136 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:16.136 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:16.136 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:15 smithi002 ceph-mon[29154]: from='osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-16T22:16:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:15 smithi103 ceph-mon[36259]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:16.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:15 smithi103 ceph-mon[36259]: from='osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-16T22:16:16.671 DEBUG:teuthology.orchestra.run.smithi002:osd.2> sudo journalctl -f -n 0 -u ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.2.service 2024-02-16T22:16:16.673 INFO:tasks.cephadm:Deploying osd.3 on smithi103 with /dev/vg_nvme/lv_4... 2024-02-16T22:16:16.673 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-16T22:16:16.933 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi103/config 2024-02-16T22:16:17.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:17 smithi002 ceph-mon[29154]: from='osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-16T22:16:17.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:17 smithi002 ceph-mon[29154]: osdmap e17: 3 total, 2 up, 3 in 2024-02-16T22:16:17.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:17 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:17.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:17 smithi002 ceph-mon[29154]: from='osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-16T22:16:17.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:17 smithi002 ceph-mon[29154]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:16:17.385+0000 7f2694eea700 -1 osd.2 0 waiting for initial osdmap 2024-02-16T22:16:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:16:17.391+0000 7f26904e6700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-16T22:16:17.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:17 smithi103 ceph-mon[36259]: from='osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-16T22:16:17.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:17 smithi103 ceph-mon[36259]: osdmap e17: 3 total, 2 up, 3 in 2024-02-16T22:16:17.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:17 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:17.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:17 smithi103 ceph-mon[36259]: from='osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-02-16T22:16:17.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:17 smithi103 ceph-mon[36259]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:18.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:18 smithi103 ceph-mon[36259]: from='osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-16T22:16:18.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:18 smithi103 ceph-mon[36259]: osdmap e18: 3 total, 2 up, 3 in 2024-02-16T22:16:18.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:18 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:18.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:18 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:18.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:18 smithi002 ceph-mon[29154]: from='osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-02-16T22:16:18.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:18 smithi002 ceph-mon[29154]: osdmap e18: 3 total, 2 up, 3 in 2024-02-16T22:16:18.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:18 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:18.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:18 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:18.767 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-16T22:16:18.767 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-16T22:16:18.767 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-02-16T22:16:18.767 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-02-16T22:16:18.768 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113703 s, 922 MB/s 2024-02-16T22:16:18.768 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-02-16T22:16:19.354 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_4 2024-02-16T22:16:19.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:19 smithi002 ceph-mon[29154]: purged_snaps scrub starts 2024-02-16T22:16:19.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:19 smithi002 ceph-mon[29154]: purged_snaps scrub ok 2024-02-16T22:16:19.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:19 smithi002 ceph-mon[29154]: osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] boot 2024-02-16T22:16:19.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:19 smithi002 ceph-mon[29154]: osdmap e19: 3 total, 3 up, 3 in 2024-02-16T22:16:19.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:19 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:19.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:19 smithi002 ceph-mon[29154]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:19.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:19 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-16T22:16:19.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:19 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:19.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:19 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:19.604 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi103/config 2024-02-16T22:16:19.725 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:19 smithi103 ceph-mon[36259]: purged_snaps scrub starts 2024-02-16T22:16:19.726 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:19 smithi103 ceph-mon[36259]: purged_snaps scrub ok 2024-02-16T22:16:19.726 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:19 smithi103 ceph-mon[36259]: osd.2 [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] boot 2024-02-16T22:16:19.726 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:19 smithi103 ceph-mon[36259]: osdmap e19: 3 total, 3 up, 3 in 2024-02-16T22:16:19.726 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:19 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-16T22:16:19.726 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:19 smithi103 ceph-mon[36259]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-16T22:16:19.726 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:19 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-16T22:16:19.726 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:19 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:19.726 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:19 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:20 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-16T22:16:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:20 smithi002 ceph-mon[29154]: osdmap e20: 3 total, 3 up, 3 in 2024-02-16T22:16:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:20 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-16T22:16:20.637 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:20 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-16T22:16:20.637 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:20 smithi103 ceph-mon[36259]: osdmap e20: 3 total, 3 up, 3 in 2024-02-16T22:16:20.638 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:20 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-16T22:16:21.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.110+0000 7f416682b700 1 -- 172.21.15.103:0/2599938796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160100120 msgr2=0x7f4160102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.110+0000 7f416682b700 1 --2- 172.21.15.103:0/2599938796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160100120 0x7f4160102520 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4154004940 tx=0x7f4154009e30 comp rx=0 tx=0).stop 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.111+0000 7f416682b700 1 -- 172.21.15.103:0/2599938796 shutdown_connections 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.111+0000 7f416682b700 1 --2- 172.21.15.103:0/2599938796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160100120 0x7f4160102520 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.111+0000 7f416682b700 1 --2- 172.21.15.103:0/2599938796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4160066c10 0x7f41600ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.111+0000 7f416682b700 1 -- 172.21.15.103:0/2599938796 >> 172.21.15.103:0/2599938796 conn(0x7f41600f7500 msgr2=0x7f41600f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.111+0000 7f416682b700 1 -- 172.21.15.103:0/2599938796 shutdown_connections 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.111+0000 7f416682b700 1 -- 172.21.15.103:0/2599938796 wait complete. 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.111+0000 7f416682b700 1 Processor -- start 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.111+0000 7f416682b700 1 -- start start 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f416682b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160066c10 0x7f41600fd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160066c10 0x7f41600fd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160066c10 0x7f41600fd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57982/0 (socket says 172.21.15.103:57982) 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f416682b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4160100120 0x7f41600fba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:16:21.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41600fc050 con 0x7f4160100120 2024-02-16T22:16:21.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415ffff700 1 -- 172.21.15.103:0/702846113 learned_addr learned my addr 172.21.15.103:0/702846113 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:16:21.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41600fc1c0 con 0x7f4160066c10 2024-02-16T22:16:21.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415ffff700 1 -- 172.21.15.103:0/702846113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4160100120 msgr2=0x7f41600fba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:16:21.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415ffff700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4160100120 0x7f41600fba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:21.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415ffff700 1 -- 172.21.15.103:0/702846113 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154005040 con 0x7f4160066c10 2024-02-16T22:16:21.115 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415ffff700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160066c10 0x7f41600fd5d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f415000eb40 tx=0x7f415000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:16:21.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415cff9700 1 -- 172.21.15.103:0/702846113 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4150007cc0 con 0x7f4160066c10 2024-02-16T22:16:21.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41600fc440 con 0x7f4160066c10 2024-02-16T22:16:21.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415cff9700 1 -- 172.21.15.103:0/702846113 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f415000f040 con 0x7f4160066c10 2024-02-16T22:16:21.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f415cff9700 1 -- 172.21.15.103:0/702846113 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4150009620 con 0x7f4160066c10 2024-02-16T22:16:21.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.112+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41601158f0 con 0x7f4160066c10 2024-02-16T22:16:21.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.113+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4160107a50 con 0x7f4160066c10 2024-02-16T22:16:21.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.114+0000 7f415cff9700 1 -- 172.21.15.103:0/702846113 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) v1 ==== 86563+0+0 (secure 0 0 0) 0x7f4150009780 con 0x7f4160066c10 2024-02-16T22:16:21.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.114+0000 7f415cff9700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4148069210 0x7f414806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:16:21.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.114+0000 7f415cff9700 1 -- 172.21.15.103:0/702846113 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f4150086430 con 0x7f4160066c10 2024-02-16T22:16:21.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.114+0000 7f415f7fe700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4148069210 0x7f414806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:16:21.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.115+0000 7f415f7fe700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4148069210 0x7f414806b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4154000c00 tx=0x7f4154009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:16:21.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.119+0000 7f415cff9700 1 -- 172.21.15.103:0/702846113 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f415004dbd0 con 0x7f4160066c10 2024-02-16T22:16:21.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.308+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f416002b410 con 0x7f4148069210 2024-02-16T22:16:21.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:21.411+0000 7f415cff9700 1 -- 172.21.15.103:0/702846113 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f4150053500 con 0x7f4160066c10 2024-02-16T22:16:21.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:21 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-16T22:16:21.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:21 smithi103 ceph-mon[36259]: osdmap e21: 3 total, 3 up, 3 in 2024-02-16T22:16:21.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:21 smithi103 ceph-mon[36259]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:21.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:21 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:16:21.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:21 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:16:21.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:21 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:21.637 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67058]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-16T22:16:21.637 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67058]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:21.637 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67058]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:21.637 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67058]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:21.638 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:21 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-16T22:16:21.638 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:21 smithi002 ceph-mon[29154]: osdmap e21: 3 total, 3 up, 3 in 2024-02-16T22:16:21.638 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:21 smithi002 ceph-mon[29154]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:21.638 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:21 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:16:21.638 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:21 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:16:21.638 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:21 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:21.944 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67064]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-16T22:16:21.944 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67064]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:21.944 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67064]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:21.944 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67064]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67075]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-16T22:16:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67075]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67075]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:16:21 smithi002 sudo[67075]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:22.509 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: osdmap e22: 3 total, 3 up, 3 in 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: mgrmap e19: smithi002.resmjv(active, since 2m), standbys: smithi103.ydydbo 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:22.510 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.511 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:16:22.511 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67178]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-16T22:16:22.511 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67178]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:22.511 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67178]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:22.511 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67178]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:22.782 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67237]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-16T22:16:22.783 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67237]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:22.783 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67237]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:22.783 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67237]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:22.790 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:16:22.790 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: osdmap e22: 3 total, 3 up, 3 in 2024-02-16T22:16:22.790 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: mgrmap e19: smithi002.resmjv(active, since 2m), standbys: smithi103.ydydbo 2024-02-16T22:16:22.790 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.790 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.790 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.790 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:16:22.791 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:16:22.791 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:16:22.791 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.791 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:22.791 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:22.791 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:22.791 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:16:23.043 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67273]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-16T22:16:23.043 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67273]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:23.043 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67273]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:23.043 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:16:22 smithi002 sudo[67273]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:23.317 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 sudo[67290]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-16T22:16:23.317 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 sudo[67290]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:23.317 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 sudo[67290]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:23.317 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 sudo[67290]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:23.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 sudo[67343]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-16T22:16:23.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 sudo[67343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:23.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 sudo[67343]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:23.575 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 sudo[67343]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:23.818 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 sudo[39612]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-16T22:16:23.818 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 sudo[39612]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:23.818 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 sudo[39612]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:23.818 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 sudo[39612]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:24.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 sudo[39615]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 sudo[39615]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 sudo[39615]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 sudo[39615]: pam_unix(sudo:session): session closed for user root 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 ceph-mon[36259]: Detected new or changed devices on smithi002 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 ceph-mon[36259]: Adjusting osd_memory_target on smithi002 to 4289M 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 ceph-mon[36259]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 ceph-mon[36259]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 ceph-mon[36259]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:16:24.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:16:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 ceph-mon[29154]: Detected new or changed devices on smithi002 2024-02-16T22:16:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 ceph-mon[29154]: Adjusting osd_memory_target on smithi002 to 4289M 2024-02-16T22:16:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 ceph-mon[29154]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 ceph-mon[29154]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-16T22:16:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 ceph-mon[29154]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-16T22:16:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:16:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:16:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi002"}]: dispatch 2024-02-16T22:16:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mon metadata", "id": "smithi103"}]: dispatch 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/186487964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a182db-e441-45f8-9d13-61d69b7db54d"}]: dispatch 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a182db-e441-45f8-9d13-61d69b7db54d"}]: dispatch 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09a182db-e441-45f8-9d13-61d69b7db54d"}]': finished 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: osdmap e23: 4 total, 3 up, 4 in 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:25.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:24 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/422435117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/186487964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a182db-e441-45f8-9d13-61d69b7db54d"}]: dispatch 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09a182db-e441-45f8-9d13-61d69b7db54d"}]: dispatch 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09a182db-e441-45f8-9d13-61d69b7db54d"}]': finished 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: osdmap e23: 4 total, 3 up, 4 in 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:25.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:24 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/422435117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:16:25.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:25 smithi002 ceph-mon[29154]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:26.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:25 smithi103 ceph-mon[36259]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:27.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:27.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:27.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:27 smithi103 ceph-mon[36259]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:27.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:27.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:27.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:27.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:27.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:27.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:27 smithi002 ceph-mon[29154]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:27.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:27.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:27.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:29.798 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:29 smithi103 ceph-mon[36259]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:29.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:29 smithi002 ceph-mon[29154]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:32.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:31 smithi103 ceph-mon[36259]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:32.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:31 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-16T22:16:32.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:31 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:32.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:31 smithi002 ceph-mon[29154]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:32.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:31 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-16T22:16:32.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:31 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:33.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:32 smithi103 ceph-mon[36259]: Deploying daemon osd.3 on smithi103 2024-02-16T22:16:33.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:32 smithi002 ceph-mon[29154]: Deploying daemon osd.3 on smithi103 2024-02-16T22:16:33.864 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:33 smithi103 ceph-mon[36259]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:34.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:33 smithi002 ceph-mon[29154]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:36.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:35 smithi002 ceph-mon[29154]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:36.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:35 smithi103 ceph-mon[36259]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:37.420 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:37 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:37.421 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:37 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:37.421 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:37 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:16:37.421 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:37 smithi103 ceph-mon[36259]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:37.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:37 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:37.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:37 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:37.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:37 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:16:37.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:37 smithi002 ceph-mon[29154]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:38.748 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:16:38.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:16:39.743 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:39 smithi103 ceph-mon[36259]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:39.824 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 3 on host 'smithi103' 2024-02-16T22:16:39.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.822+0000 7f415cff9700 1 -- 172.21.15.103:0/702846113 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f416002b410 con 0x7f4148069210 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4148069210 msgr2=0x7f414806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4148069210 0x7f414806b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4154000c00 tx=0x7f4154009e30 comp rx=0 tx=0).stop 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160066c10 msgr2=0x7f41600fd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160066c10 0x7f41600fd5d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f415000eb40 tx=0x7f415000b040 comp rx=0 tx=0).stop 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 shutdown_connections 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4148069210 0x7f414806b6c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4160066c10 0x7f41600fd5d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 --2- 172.21.15.103:0/702846113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4160100120 0x7f41600fba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 >> 172.21.15.103:0/702846113 conn(0x7f41600f7500 msgr2=0x7f41600f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 shutdown_connections 2024-02-16T22:16:39.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:39.824+0000 7f416682b700 1 -- 172.21.15.103:0/702846113 wait complete. 2024-02-16T22:16:39.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:39 smithi002 ceph-mon[29154]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:40.362 DEBUG:teuthology.orchestra.run.smithi103:osd.3> sudo journalctl -f -n 0 -u ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.3.service 2024-02-16T22:16:40.364 INFO:tasks.cephadm:Deploying osd.4 on smithi103 with /dev/vg_nvme/lv_3... 2024-02-16T22:16:40.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-16T22:16:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi103/config 2024-02-16T22:16:41.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:40 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:41.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:40 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:41.023 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:40 smithi103 ceph-mon[36259]: from='osd.3 [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-16T22:16:41.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:40 smithi103 ceph-mon[36259]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-16T22:16:41.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:40 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:41.024 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:40 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:41.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:40 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:41.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:40 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:41.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:40 smithi002 ceph-mon[29154]: from='osd.3 [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-16T22:16:41.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:40 smithi002 ceph-mon[29154]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-16T22:16:41.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:40 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:41.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:40 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:42.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:41 smithi103 ceph-mon[36259]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:42.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:41 smithi103 ceph-mon[36259]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-16T22:16:42.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:41 smithi103 ceph-mon[36259]: osdmap e24: 4 total, 3 up, 4 in 2024-02-16T22:16:42.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:41 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:42.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:41 smithi103 ceph-mon[36259]: from='osd.3 [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:16:42.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:41 smithi103 ceph-mon[36259]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:16:42.076 INFO:journalctl@ceph.osd.3.smithi103.stdout:Feb 16 22:16:41 smithi103 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-3[41690]: 2024-02-16T22:16:41.865+0000 7f2c4c06c700 -1 osd.3 0 waiting for initial osdmap 2024-02-16T22:16:42.076 INFO:journalctl@ceph.osd.3.smithi103.stdout:Feb 16 22:16:41 smithi103 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-3[41690]: 2024-02-16T22:16:41.871+0000 7f2c48e6b700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-16T22:16:42.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:41 smithi002 ceph-mon[29154]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:42.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:41 smithi002 ceph-mon[29154]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-16T22:16:42.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:41 smithi002 ceph-mon[29154]: osdmap e24: 4 total, 3 up, 4 in 2024-02-16T22:16:42.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:41 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:42.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:41 smithi002 ceph-mon[29154]: from='osd.3 [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:16:42.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:41 smithi002 ceph-mon[29154]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:16:43.175 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-16T22:16:43.175 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-16T22:16:43.176 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-02-16T22:16:43.176 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-02-16T22:16:43.176 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122792 s, 854 MB/s 2024-02-16T22:16:43.176 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-02-16T22:16:43.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:42 smithi103 ceph-mon[36259]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-02-16T22:16:43.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:42 smithi103 ceph-mon[36259]: osdmap e25: 4 total, 3 up, 4 in 2024-02-16T22:16:43.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:42 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:43.176 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:42 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:43.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:42 smithi002 ceph-mon[29154]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-02-16T22:16:43.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:42 smithi002 ceph-mon[29154]: osdmap e25: 4 total, 3 up, 4 in 2024-02-16T22:16:43.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:42 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:43.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:42 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:43.760 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_3 2024-02-16T22:16:43.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:43 smithi103 ceph-mon[36259]: purged_snaps scrub starts 2024-02-16T22:16:43.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:43 smithi103 ceph-mon[36259]: purged_snaps scrub ok 2024-02-16T22:16:43.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:43 smithi103 ceph-mon[36259]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:43.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:43.993 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:43 smithi103 ceph-mon[36259]: osd.3 [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] boot 2024-02-16T22:16:43.994 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:43 smithi103 ceph-mon[36259]: osdmap e26: 4 total, 4 up, 4 in 2024-02-16T22:16:43.994 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:43 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:44.225 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi103/config 2024-02-16T22:16:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:43 smithi002 ceph-mon[29154]: purged_snaps scrub starts 2024-02-16T22:16:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:43 smithi002 ceph-mon[29154]: purged_snaps scrub ok 2024-02-16T22:16:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:43 smithi002 ceph-mon[29154]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-16T22:16:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:43 smithi002 ceph-mon[29154]: osd.3 [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] boot 2024-02-16T22:16:44.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:43 smithi002 ceph-mon[29154]: osdmap e26: 4 total, 4 up, 4 in 2024-02-16T22:16:44.227 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:43 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-16T22:16:45.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:44 smithi103 ceph-mon[36259]: osdmap e27: 4 total, 4 up, 4 in 2024-02-16T22:16:45.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:44 smithi103 ceph-mon[36259]: osdmap e28: 4 total, 4 up, 4 in 2024-02-16T22:16:45.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:44 smithi002 ceph-mon[29154]: osdmap e27: 4 total, 4 up, 4 in 2024-02-16T22:16:45.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:44 smithi002 ceph-mon[29154]: osdmap e28: 4 total, 4 up, 4 in 2024-02-16T22:16:46.156 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-16T22:16:46.156 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: Detected new or changed devices on smithi103 2024-02-16T22:16:46.156 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.156 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.156 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:16:46.157 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: Adjusting osd_memory_target on smithi103 to 15936M 2024-02-16T22:16:46.157 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.157 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:46.157 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:46.157 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.157 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:16:46.157 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:46.157 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:46.157 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-16T22:16:46.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: Detected new or changed devices on smithi103 2024-02-16T22:16:46.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: Adjusting osd_memory_target on smithi103 to 15936M 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:16:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:16:46.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.343+0000 7f32eb9eb700 1 -- 172.21.15.103:0/1557835344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 msgr2=0x7f32e40fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:16:46.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 --2- 172.21.15.103:0/1557835344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 0x7f32e40fe550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f32d4004970 tx=0x7f32d4009e30 comp rx=0 tx=0).stop 2024-02-16T22:16:46.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 -- 172.21.15.103:0/1557835344 shutdown_connections 2024-02-16T22:16:46.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 --2- 172.21.15.103:0/1557835344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 0x7f32e40fe550 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 --2- 172.21.15.103:0/1557835344 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e40fcef0 0x7f32e40fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 -- 172.21.15.103:0/1557835344 >> 172.21.15.103:0/1557835344 conn(0x7f32e40f8620 msgr2=0x7f32e40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 -- 172.21.15.103:0/1557835344 shutdown_connections 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 -- 172.21.15.103:0/1557835344 wait complete. 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 Processor -- start 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 -- start start 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e40fcef0 0x7f32e410fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 0x7f32e4110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e4110620 con 0x7f32e40fcef0 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32eb9eb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e4110ec0 con 0x7f32e40fe0e0 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32e8f86700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 0x7f32e4110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:16:46.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32e8f86700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 0x7f32e4110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46688/0 (socket says 172.21.15.103:46688) 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.344+0000 7f32e8f86700 1 -- 172.21.15.103:0/969494198 learned_addr learned my addr 172.21.15.103:0/969494198 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32e9787700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e40fcef0 0x7f32e410fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32e8f86700 1 -- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e40fcef0 msgr2=0x7f32e410fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32e8f86700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e40fcef0 0x7f32e410fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32e8f86700 1 -- 172.21.15.103:0/969494198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4005040 con 0x7f32e40fe0e0 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32e8f86700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 0x7f32e4110020 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f32d400b370 tx=0x7f32d4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32d9ffb700 1 -- 172.21.15.103:0/969494198 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32d400e040 con 0x7f32e40fe0e0 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e4111140 con 0x7f32e40fe0e0 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32d9ffb700 1 -- 172.21.15.103:0/969494198 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32d400bd60 con 0x7f32e40fe0e0 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32d9ffb700 1 -- 172.21.15.103:0/969494198 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32d401c630 con 0x7f32e40fe0e0 2024-02-16T22:16:46.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.345+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e4111680 con 0x7f32e40fe0e0 2024-02-16T22:16:46.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.346+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e4102210 con 0x7f32e40fe0e0 2024-02-16T22:16:46.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.347+0000 7f32d9ffb700 1 -- 172.21.15.103:0/969494198 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f32d4013a80 con 0x7f32e40fe0e0 2024-02-16T22:16:46.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.347+0000 7f32d9ffb700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32d00692a0 0x7f32d006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:16:46.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.347+0000 7f32d9ffb700 1 -- 172.21.15.103:0/969494198 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f32d4087f60 con 0x7f32e40fe0e0 2024-02-16T22:16:46.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.347+0000 7f32e9787700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32d00692a0 0x7f32d006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:16:46.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.348+0000 7f32e9787700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32d00692a0 0x7f32d006b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32e40fcc30 tx=0x7f32e000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:16:46.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.351+0000 7f32d9ffb700 1 -- 172.21.15.103:0/969494198 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f32d40b9bd0 con 0x7f32e40fe0e0 2024-02-16T22:16:46.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:16:46.543+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f32e41119d0 con 0x7f32d00692a0 2024-02-16T22:16:47.149 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:46 smithi103 ceph-mon[36259]: osdmap e29: 4 total, 4 up, 4 in 2024-02-16T22:16:47.149 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:46 smithi103 ceph-mon[36259]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:16:47.149 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:46 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:16:47.149 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:46 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:16:47.149 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:46 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:47.149 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:46 smithi103 ceph-mon[36259]: pgmap v96: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 1/6 objects misplaced (16.667%); 64 KiB/s, 0 objects/s recovering 2024-02-16T22:16:47.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:46 smithi002 ceph-mon[29154]: osdmap e29: 4 total, 4 up, 4 in 2024-02-16T22:16:47.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:46 smithi002 ceph-mon[29154]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:16:47.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:46 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:16:47.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:46 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:16:47.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:46 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:47.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:46 smithi002 ceph-mon[29154]: pgmap v96: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 1/6 objects misplaced (16.667%); 64 KiB/s, 0 objects/s recovering 2024-02-16T22:16:48.040 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:47 smithi103 ceph-mon[36259]: osdmap e30: 4 total, 4 up, 4 in 2024-02-16T22:16:48.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:47 smithi002 ceph-mon[29154]: osdmap e30: 4 total, 4 up, 4 in 2024-02-16T22:16:49.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:48 smithi002 ceph-mon[29154]: pgmap v98: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 1/6 objects misplaced (16.667%); 55 KiB/s, 0 objects/s recovering 2024-02-16T22:16:49.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:48 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/4202366670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee9a7426-801d-420f-be92-cd18c21d11b7"}]: dispatch 2024-02-16T22:16:49.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:48 smithi002 ceph-mon[29154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee9a7426-801d-420f-be92-cd18c21d11b7"}]: dispatch 2024-02-16T22:16:49.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:48 smithi002 ceph-mon[29154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee9a7426-801d-420f-be92-cd18c21d11b7"}]': finished 2024-02-16T22:16:49.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:48 smithi002 ceph-mon[29154]: osdmap e31: 5 total, 4 up, 5 in 2024-02-16T22:16:49.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:48 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:16:49.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:48 smithi103 ceph-mon[36259]: pgmap v98: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 1/6 objects misplaced (16.667%); 55 KiB/s, 0 objects/s recovering 2024-02-16T22:16:49.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:48 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/4202366670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee9a7426-801d-420f-be92-cd18c21d11b7"}]: dispatch 2024-02-16T22:16:49.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:48 smithi103 ceph-mon[36259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee9a7426-801d-420f-be92-cd18c21d11b7"}]: dispatch 2024-02-16T22:16:49.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:48 smithi103 ceph-mon[36259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee9a7426-801d-420f-be92-cd18c21d11b7"}]': finished 2024-02-16T22:16:49.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:48 smithi103 ceph-mon[36259]: osdmap e31: 5 total, 4 up, 5 in 2024-02-16T22:16:49.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:48 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:16:50.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:49 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/1698043225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:16:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:49 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/1698043225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:16:51.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:50 smithi002 ceph-mon[29154]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-02-16T22:16:51.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:50 smithi103 ceph-mon[36259]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-02-16T22:16:53.890 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:53 smithi103 ceph-mon[36259]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-02-16T22:16:53.890 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:16:53.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:53 smithi002 ceph-mon[29154]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-02-16T22:16:53.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:16:56.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:55 smithi103 ceph-mon[36259]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-02-16T22:16:56.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-16T22:16:56.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:55 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:55 smithi002 ceph-mon[29154]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-02-16T22:16:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-16T22:16:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:55 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:16:57.008 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:56 smithi002 ceph-mon[29154]: Deploying daemon osd.4 on smithi103 2024-02-16T22:16:57.244 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:56 smithi103 ceph-mon[36259]: Deploying daemon osd.4 on smithi103 2024-02-16T22:16:58.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:57 smithi103 ceph-mon[36259]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 33 KiB/s, 0 objects/s recovering 2024-02-16T22:16:58.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:57 smithi002 ceph-mon[29154]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 33 KiB/s, 0 objects/s recovering 2024-02-16T22:16:59.958 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:16:59 smithi103 ceph-mon[36259]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-02-16T22:17:00.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:16:59 smithi002 ceph-mon[29154]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-02-16T22:17:02.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:01 smithi103 ceph-mon[36259]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 27 KiB/s, 0 objects/s recovering 2024-02-16T22:17:02.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:01 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:02.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:01 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:02.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:01 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:17:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:01 smithi002 ceph-mon[29154]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 27 KiB/s, 0 objects/s recovering 2024-02-16T22:17:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:01 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:01 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:01 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:17:04.074 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:03 smithi103 ceph-mon[36259]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-16T22:17:04.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:03 smithi002 ceph-mon[29154]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-16T22:17:04.407 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 4 on host 'smithi103' 2024-02-16T22:17:04.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.404+0000 7f32d9ffb700 1 -- 172.21.15.103:0/969494198 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f32e41119d0 con 0x7f32d00692a0 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32d00692a0 msgr2=0x7f32d006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32d00692a0 0x7f32d006b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32e40fcc30 tx=0x7f32e000b040 comp rx=0 tx=0).stop 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 msgr2=0x7f32e4110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 0x7f32e4110020 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f32d400b370 tx=0x7f32d4009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 shutdown_connections 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32d00692a0 0x7f32d006b750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e40fcef0 0x7f32e410fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 --2- 172.21.15.103:0/969494198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e40fe0e0 0x7f32e4110020 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 >> 172.21.15.103:0/969494198 conn(0x7f32e40f8620 msgr2=0x7f32e40ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 shutdown_connections 2024-02-16T22:17:04.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:04.407+0000 7f32eb9eb700 1 -- 172.21.15.103:0/969494198 wait complete. 2024-02-16T22:17:04.985 DEBUG:teuthology.orchestra.run.smithi103:osd.4> sudo journalctl -f -n 0 -u ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.4.service 2024-02-16T22:17:04.988 INFO:tasks.cephadm:Deploying osd.5 on smithi103 with /dev/vg_nvme/lv_2... 2024-02-16T22:17:04.988 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-16T22:17:05.306 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi103/config 2024-02-16T22:17:05.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:05 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:05.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:05 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:05.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:05 smithi103 ceph-mon[36259]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-16T22:17:05.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:05 smithi103 ceph-mon[36259]: from='osd.4 [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-16T22:17:05.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:05 smithi103 ceph-mon[36259]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-16T22:17:05.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:05 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:05.622 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:05 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:05.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:05 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:05.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:05 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:05.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:05 smithi002 ceph-mon[29154]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-16T22:17:05.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:05 smithi002 ceph-mon[29154]: from='osd.4 [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-16T22:17:05.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:05 smithi002 ceph-mon[29154]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-16T22:17:05.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:05 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:05.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:05 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:06.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:06 smithi103 ceph-mon[36259]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-16T22:17:06.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:06 smithi103 ceph-mon[36259]: osdmap e32: 5 total, 4 up, 5 in 2024-02-16T22:17:06.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:06 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:17:06.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:06 smithi103 ceph-mon[36259]: from='osd.4 [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:17:06.621 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:06 smithi103 ceph-mon[36259]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:17:06.621 INFO:journalctl@ceph.osd.4.smithi103.stdout:Feb 16 22:17:06 smithi103 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-4[48107]: 2024-02-16T22:17:06.382+0000 7f4f4bf82700 -1 osd.4 0 waiting for initial osdmap 2024-02-16T22:17:06.621 INFO:journalctl@ceph.osd.4.smithi103.stdout:Feb 16 22:17:06 smithi103 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-4[48107]: 2024-02-16T22:17:06.389+0000 7f4f4557a700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-16T22:17:06.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:06 smithi002 ceph-mon[29154]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-16T22:17:06.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:06 smithi002 ceph-mon[29154]: osdmap e32: 5 total, 4 up, 5 in 2024-02-16T22:17:06.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:06 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:17:06.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:06 smithi002 ceph-mon[29154]: from='osd.4 [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:17:06.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:06 smithi002 ceph-mon[29154]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:17:07.443 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-16T22:17:07.443 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-16T22:17:07.443 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-02-16T22:17:07.443 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-02-16T22:17:07.443 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122549 s, 856 MB/s 2024-02-16T22:17:07.444 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-02-16T22:17:07.696 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:07 smithi103 ceph-mon[36259]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-02-16T22:17:07.696 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:07 smithi103 ceph-mon[36259]: osdmap e33: 5 total, 4 up, 5 in 2024-02-16T22:17:07.696 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:17:07.696 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:17:07.696 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:07 smithi103 ceph-mon[36259]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-16T22:17:07.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:07 smithi002 ceph-mon[29154]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-02-16T22:17:07.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:07 smithi002 ceph-mon[29154]: osdmap e33: 5 total, 4 up, 5 in 2024-02-16T22:17:07.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:17:07.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:17:07.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:07 smithi002 ceph-mon[29154]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-16T22:17:08.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:08 smithi103 ceph-mon[36259]: purged_snaps scrub starts 2024-02-16T22:17:08.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:08 smithi103 ceph-mon[36259]: purged_snaps scrub ok 2024-02-16T22:17:08.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:08 smithi103 ceph-mon[36259]: osd.4 [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] boot 2024-02-16T22:17:08.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:08 smithi103 ceph-mon[36259]: osdmap e34: 5 total, 5 up, 5 in 2024-02-16T22:17:08.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:08 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:17:08.600 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_2 2024-02-16T22:17:08.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:08 smithi002 ceph-mon[29154]: purged_snaps scrub starts 2024-02-16T22:17:08.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:08 smithi002 ceph-mon[29154]: purged_snaps scrub ok 2024-02-16T22:17:08.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:08 smithi002 ceph-mon[29154]: osd.4 [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] boot 2024-02-16T22:17:08.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:08 smithi002 ceph-mon[29154]: osdmap e34: 5 total, 5 up, 5 in 2024-02-16T22:17:08.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-16T22:17:08.932 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi103/config 2024-02-16T22:17:09.483 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:09 smithi103 ceph-mon[36259]: osdmap e35: 5 total, 5 up, 5 in 2024-02-16T22:17:09.483 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:09 smithi103 ceph-mon[36259]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:09.483 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:09 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:17:09.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:09 smithi002 ceph-mon[29154]: osdmap e35: 5 total, 5 up, 5 in 2024-02-16T22:17:09.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:09 smithi002 ceph-mon[29154]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:09.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:17:11.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.150+0000 7f5d9177f700 1 -- 172.21.15.103:0/3593587870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 msgr2=0x7f5d8c070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.150+0000 7f5d9177f700 1 --2- 172.21.15.103:0/3593587870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 0x7f5d8c070300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c004940 tx=0x7f5d7c009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.150+0000 7f5d9177f700 1 -- 172.21.15.103:0/3593587870 shutdown_connections 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.150+0000 7f5d9177f700 1 --2- 172.21.15.103:0/3593587870 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d8c070840 0x7f5d8c070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.150+0000 7f5d9177f700 1 --2- 172.21.15.103:0/3593587870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 0x7f5d8c070300 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.150+0000 7f5d9177f700 1 -- 172.21.15.103:0/3593587870 >> 172.21.15.103:0/3593587870 conn(0x7f5d8c0f8620 msgr2=0x7f5d8c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.150+0000 7f5d9177f700 1 -- 172.21.15.103:0/3593587870 shutdown_connections 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.150+0000 7f5d9177f700 1 -- 172.21.15.103:0/3593587870 wait complete. 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.150+0000 7f5d9177f700 1 Processor -- start 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d9177f700 1 -- start start 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d9177f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d8c070840 0x7f5d8c10fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d9177f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 0x7f5d8c110110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d9177f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c110710 con 0x7f5d8c070840 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d9177f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c110fb0 con 0x7f5d8c071e90 2024-02-16T22:17:11.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d827fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 0x7f5d8c110110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d827fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 0x7f5d8c110110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52866/0 (socket says 172.21.15.103:52866) 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d827fc700 1 -- 172.21.15.103:0/4093689981 learned_addr learned my addr 172.21.15.103:0/4093689981 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d8affd700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d8c070840 0x7f5d8c10fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d827fc700 1 -- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d8c070840 msgr2=0x7f5d8c10fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d827fc700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d8c070840 0x7f5d8c10fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d827fc700 1 -- 172.21.15.103:0/4093689981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c005040 con 0x7f5d8c071e90 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.151+0000 7f5d827fc700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 0x7f5d8c110110 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5d74002a50 tx=0x7f5d740046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.152+0000 7f5d83fff700 1 -- 172.21.15.103:0/4093689981 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d74007cc0 con 0x7f5d8c071e90 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.152+0000 7f5d83fff700 1 -- 172.21.15.103:0/4093689981 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d7400f040 con 0x7f5d8c071e90 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.152+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c111290 con 0x7f5d8c071e90 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.152+0000 7f5d83fff700 1 -- 172.21.15.103:0/4093689981 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d7400da30 con 0x7f5d8c071e90 2024-02-16T22:17:11.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.152+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c1117d0 con 0x7f5d8c071e90 2024-02-16T22:17:11.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.153+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8c04bed0 con 0x7f5d8c071e90 2024-02-16T22:17:11.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.153+0000 7f5d83fff700 1 -- 172.21.15.103:0/4093689981 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f5d7400db90 con 0x7f5d8c071e90 2024-02-16T22:17:11.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.153+0000 7f5d83fff700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5d780692a0 0x7f5d7806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:11.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.153+0000 7f5d83fff700 1 -- 172.21.15.103:0/4093689981 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f5d74086a50 con 0x7f5d8c071e90 2024-02-16T22:17:11.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.154+0000 7f5d8affd700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5d780692a0 0x7f5d7806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:11.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.154+0000 7f5d8affd700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5d780692a0 0x7f5d7806b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c00b3b0 tx=0x7f5d7c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:11.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.158+0000 7f5d83fff700 1 -- 172.21.15.103:0/4093689981 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5d74056c80 con 0x7f5d8c071e90 2024-02-16T22:17:11.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:11.352+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d8c111ba0 con 0x7f5d780692a0 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: Detected new or changed devices on smithi103 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: Adjusting osd_memory_target on smithi103 to 7968M 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:17:11.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:11.577 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:17:11.577 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.577 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:11 smithi103 ceph-mon[36259]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: Detected new or changed devices on smithi103 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: Adjusting osd_memory_target on smithi103 to 7968M 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:17:11.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:11.727 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:11 smithi002 ceph-mon[29154]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:12.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:12 smithi103 ceph-mon[36259]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:17:12.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:12 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:17:12.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:12 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:17:12.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:12 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:12.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:12 smithi002 ceph-mon[29154]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:17:12.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:12 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-16T22:17:12.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:12 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-16T22:17:12.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:12 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:13.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:13 smithi103 ceph-mon[36259]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:13.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:13 smithi002 ceph-mon[29154]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:14.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:14 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/1221738623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68c72748-2d27-4ada-a35b-53e1bab20f9d"}]: dispatch 2024-02-16T22:17:14.577 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:14 smithi103 ceph-mon[36259]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68c72748-2d27-4ada-a35b-53e1bab20f9d"}]: dispatch 2024-02-16T22:17:14.577 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:14 smithi103 ceph-mon[36259]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68c72748-2d27-4ada-a35b-53e1bab20f9d"}]': finished 2024-02-16T22:17:14.577 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:14 smithi103 ceph-mon[36259]: osdmap e36: 6 total, 5 up, 6 in 2024-02-16T22:17:14.577 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:14 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:14.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:14 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/1221738623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68c72748-2d27-4ada-a35b-53e1bab20f9d"}]: dispatch 2024-02-16T22:17:14.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:14 smithi002 ceph-mon[29154]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68c72748-2d27-4ada-a35b-53e1bab20f9d"}]: dispatch 2024-02-16T22:17:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:14 smithi002 ceph-mon[29154]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68c72748-2d27-4ada-a35b-53e1bab20f9d"}]': finished 2024-02-16T22:17:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:14 smithi002 ceph-mon[29154]: osdmap e36: 6 total, 5 up, 6 in 2024-02-16T22:17:14.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:14 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:15.575 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:15 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/1262348830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:17:15.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:15 smithi103 ceph-mon[36259]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:15.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:15 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/1262348830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-16T22:17:15.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:15 smithi002 ceph-mon[29154]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:17.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:17 smithi002 ceph-mon[29154]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:18.034 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:17 smithi103 ceph-mon[36259]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:19.699 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:19 smithi103 ceph-mon[36259]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:19.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:19 smithi002 ceph-mon[29154]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:22.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:21 smithi002 ceph-mon[29154]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:22.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:21 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-16T22:17:22.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:21 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:21 smithi103 ceph-mon[36259]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:21 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-16T22:17:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:21 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:23.046 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:22 smithi103 ceph-mon[36259]: Deploying daemon osd.5 on smithi103 2024-02-16T22:17:23.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:22 smithi002 ceph-mon[29154]: Deploying daemon osd.5 on smithi103 2024-02-16T22:17:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:23 smithi002 ceph-mon[29154]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:17:24.256 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:23 smithi103 ceph-mon[36259]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:24.256 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:17:25.918 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:25 smithi103 ceph-mon[36259]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:26.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:25 smithi002 ceph-mon[29154]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:27.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:27.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:27.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:27 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:17:27.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:27 smithi103 ceph-mon[36259]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:27.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:27.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:27.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:27 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:17:27.476 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:27 smithi002 ceph-mon[29154]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:29.719 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:29 smithi103 ceph-mon[36259]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:29.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.772+0000 7f5d83fff700 1 -- 172.21.15.103:0/4093689981 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5d8c111ba0 con 0x7f5d780692a0 2024-02-16T22:17:29.774 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 5 on host 'smithi103' 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5d780692a0 msgr2=0x7f5d7806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5d780692a0 0x7f5d7806b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c00b3b0 tx=0x7f5d7c009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 msgr2=0x7f5d8c110110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 0x7f5d8c110110 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5d74002a50 tx=0x7f5d740046b0 comp rx=0 tx=0).stop 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 shutdown_connections 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5d780692a0 0x7f5d7806b750 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d8c070840 0x7f5d8c10fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 --2- 172.21.15.103:0/4093689981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d8c071e90 0x7f5d8c110110 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 >> 172.21.15.103:0/4093689981 conn(0x7f5d8c0f8620 msgr2=0x7f5d8c102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 shutdown_connections 2024-02-16T22:17:29.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:29.774+0000 7f5d9177f700 1 -- 172.21.15.103:0/4093689981 wait complete. 2024-02-16T22:17:29.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:29 smithi002 ceph-mon[29154]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:30.383 DEBUG:teuthology.orchestra.run.smithi103:osd.5> sudo journalctl -f -n 0 -u ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.5.service 2024-02-16T22:17:30.385 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-16T22:17:30.385 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd stat -f json 2024-02-16T22:17:30.660 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:31.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:30 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:31.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:30 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:31.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:30 smithi103 ceph-mon[36259]: from='osd.5 [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-16T22:17:31.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:30 smithi103 ceph-mon[36259]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-16T22:17:31.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:30 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:31.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:30 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:31.077 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:30 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:31.077 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:30 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:31.077 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:30 smithi002 ceph-mon[29154]: from='osd.5 [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-16T22:17:31.077 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:30 smithi002 ceph-mon[29154]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-16T22:17:31.077 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:30 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:31.077 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:30 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:31.970 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:31 smithi002 ceph-mon[29154]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:31.970 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:31 smithi002 ceph-mon[29154]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-16T22:17:31.970 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:31 smithi002 ceph-mon[29154]: osdmap e37: 6 total, 5 up, 6 in 2024-02-16T22:17:31.970 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:31 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:31.971 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:31 smithi002 ceph-mon[29154]: from='osd.5 [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:17:31.971 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:31 smithi002 ceph-mon[29154]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:17:32.130 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:31 smithi103 ceph-mon[36259]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:32.130 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:31 smithi103 ceph-mon[36259]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-16T22:17:32.130 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:31 smithi103 ceph-mon[36259]: osdmap e37: 6 total, 5 up, 6 in 2024-02-16T22:17:32.130 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:31 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:32.130 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:31 smithi103 ceph-mon[36259]: from='osd.5 [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:17:32.130 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:31 smithi103 ceph-mon[36259]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-02-16T22:17:32.130 INFO:journalctl@ceph.osd.5.smithi103.stdout:Feb 16 22:17:31 smithi103 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-5[55028]: 2024-02-16T22:17:31.873+0000 7f00346eb700 -1 osd.5 0 waiting for initial osdmap 2024-02-16T22:17:32.130 INFO:journalctl@ceph.osd.5.smithi103.stdout:Feb 16 22:17:31 smithi103 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-5[55028]: 2024-02-16T22:17:31.881+0000 7f002dce3700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-16T22:17:32.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.200+0000 7f0e347a7700 1 -- 172.21.15.2:0/3031390096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 msgr2=0x7f0e2c0ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:32.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.200+0000 7f0e347a7700 1 --2- 172.21.15.2:0/3031390096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 0x7f0e2c0ffe60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0e28004940 tx=0x7f0e28009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:32.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 -- 172.21.15.2:0/3031390096 shutdown_connections 2024-02-16T22:17:32.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 --2- 172.21.15.2:0/3031390096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e2c1003a0 0x7f0e2c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 --2- 172.21.15.2:0/3031390096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 0x7f0e2c0ffe60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 -- 172.21.15.2:0/3031390096 >> 172.21.15.2:0/3031390096 conn(0x7f0e2c0f7520 msgr2=0x7f0e2c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 -- 172.21.15.2:0/3031390096 shutdown_connections 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 -- 172.21.15.2:0/3031390096 wait complete. 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 Processor -- start 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 -- start start 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 0x7f0e2c10d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e2c1003a0 0x7f0e2c10dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c10e3a0 con 0x7f0e2c0fda80 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e347a7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c10ec40 con 0x7f0e2c1003a0 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e32543700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 0x7f0e2c10d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:32.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e32543700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 0x7f0e2c10d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52236/0 (socket says 172.21.15.2:52236) 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.201+0000 7f0e32543700 1 -- 172.21.15.2:0/3978928310 learned_addr learned my addr 172.21.15.2:0/3978928310 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e31d42700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e2c1003a0 0x7f0e2c10dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e32543700 1 -- 172.21.15.2:0/3978928310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e2c1003a0 msgr2=0x7f0e2c10dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e32543700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e2c1003a0 0x7f0e2c10dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e32543700 1 -- 172.21.15.2:0/3978928310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28005040 con 0x7f0e2c0fda80 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e32543700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 0x7f0e2c10d860 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0e28000c00 tx=0x7f0e28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e22ffd700 1 -- 172.21.15.2:0/3978928310 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e28021070 con 0x7f0e2c0fda80 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e22ffd700 1 -- 172.21.15.2:0/3978928310 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e2801c400 con 0x7f0e2c0fda80 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e22ffd700 1 -- 172.21.15.2:0/3978928310 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2800e070 con 0x7f0e2c0fda80 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c10eec0 con 0x7f0e2c0fda80 2024-02-16T22:17:32.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.202+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c10f3a0 con 0x7f0e2c0fda80 2024-02-16T22:17:32.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.203+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c107a50 con 0x7f0e2c0fda80 2024-02-16T22:17:32.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.204+0000 7f0e22ffd700 1 -- 172.21.15.2:0/3978928310 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f0e2801ca50 con 0x7f0e2c0fda80 2024-02-16T22:17:32.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.204+0000 7f0e22ffd700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0e180692a0 0x7f0e1806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:32.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.204+0000 7f0e22ffd700 1 -- 172.21.15.2:0/3978928310 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f0e28087e40 con 0x7f0e2c0fda80 2024-02-16T22:17:32.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.204+0000 7f0e31d42700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0e180692a0 0x7f0e1806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:32.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.204+0000 7f0e31d42700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0e180692a0 0x7f0e1806b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009b80 tx=0x7f0e1c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:32.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.209+0000 7f0e22ffd700 1 -- 172.21.15.2:0/3978928310 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0e280580f0 con 0x7f0e2c0fda80 2024-02-16T22:17:32.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.403+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f0e2c02b340 con 0x7f0e2c0fda80 2024-02-16T22:17:32.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.403+0000 7f0e22ffd700 1 -- 172.21.15.2:0/3978928310 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f0e28057c80 con 0x7f0e2c0fda80 2024-02-16T22:17:32.406 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:17:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0e180692a0 msgr2=0x7f0e1806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0e180692a0 0x7f0e1806b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009b80 tx=0x7f0e1c00b040 comp rx=0 tx=0).stop 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 msgr2=0x7f0e2c10d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 0x7f0e2c10d860 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0e28000c00 tx=0x7f0e28009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 shutdown_connections 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0e180692a0 0x7f0e1806b750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e2c0fda80 0x7f0e2c10d860 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 --2- 172.21.15.2:0/3978928310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0e2c1003a0 0x7f0e2c10dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 >> 172.21.15.2:0/3978928310 conn(0x7f0e2c0f7520 msgr2=0x7f0e2c0f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 shutdown_connections 2024-02-16T22:17:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:32.406+0000 7f0e347a7700 1 -- 172.21.15.2:0/3978928310 wait complete. 2024-02-16T22:17:33.036 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:32 smithi002 ceph-mon[29154]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-02-16T22:17:33.036 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:32 smithi002 ceph-mon[29154]: osdmap e38: 6 total, 5 up, 6 in 2024-02-16T22:17:33.036 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:32 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:33.036 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:32 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:33.036 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:32 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3978928310' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-16T22:17:33.037 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1708121827,"num_in_osds":6,"osd_in_since":1708121833,"num_remapped_pgs":0} 2024-02-16T22:17:33.183 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:32 smithi103 ceph-mon[36259]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-02-16T22:17:33.184 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:32 smithi103 ceph-mon[36259]: osdmap e38: 6 total, 5 up, 6 in 2024-02-16T22:17:33.184 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:32 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:33.184 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:32 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:33.184 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:32 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3978928310' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-16T22:17:34.039 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd stat -f json 2024-02-16T22:17:34.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: purged_snaps scrub starts 2024-02-16T22:17:34.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: purged_snaps scrub ok 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: osd.5 [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] boot 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: osdmap e39: 6 total, 6 up, 6 in 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:34.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:17:34.077 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:33 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:34.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: purged_snaps scrub starts 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: purged_snaps scrub ok 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: osd.5 [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] boot 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: osdmap e39: 6 total, 6 up, 6 in 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:34.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:34.227 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:17:34.227 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:33 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:34.315 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:35.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:34 smithi002 ceph-mon[29154]: Detected new or changed devices on smithi103 2024-02-16T22:17:35.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:34 smithi002 ceph-mon[29154]: Adjusting osd_memory_target on smithi103 to 5312M 2024-02-16T22:17:35.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:34 smithi002 ceph-mon[29154]: osdmap e40: 6 total, 6 up, 6 in 2024-02-16T22:17:35.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:34 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:17:35.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:34 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:35.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:34 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:17:35.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:34 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:35.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:34 smithi103 ceph-mon[36259]: Detected new or changed devices on smithi103 2024-02-16T22:17:35.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:34 smithi103 ceph-mon[36259]: Adjusting osd_memory_target on smithi103 to 5312M 2024-02-16T22:17:35.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:34 smithi103 ceph-mon[36259]: osdmap e40: 6 total, 6 up, 6 in 2024-02-16T22:17:35.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:34 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:17:35.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:34 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:17:35.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:34 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:17:35.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:34 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:17:35.968 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:35 smithi002 ceph-mon[29154]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.983+0000 7fb69b130700 1 -- 172.21.15.2:0/3356502534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 msgr2=0x7fb6940fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.983+0000 7fb69b130700 1 --2- 172.21.15.2:0/3356502534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 0x7fb6940fe550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb680004940 tx=0x7fb680009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.983+0000 7fb69b130700 1 -- 172.21.15.2:0/3356502534 shutdown_connections 2024-02-16T22:17:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.983+0000 7fb69b130700 1 --2- 172.21.15.2:0/3356502534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 0x7fb6940fe550 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.983+0000 7fb69b130700 1 --2- 172.21.15.2:0/3356502534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6940fcef0 0x7fb6940fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.983+0000 7fb69b130700 1 -- 172.21.15.2:0/3356502534 >> 172.21.15.2:0/3356502534 conn(0x7fb6940f8620 msgr2=0x7fb6940faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb69b130700 1 -- 172.21.15.2:0/3356502534 shutdown_connections 2024-02-16T22:17:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb69b130700 1 -- 172.21.15.2:0/3356502534 wait complete. 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb69b130700 1 Processor -- start 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb69b130700 1 -- start start 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb69b130700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6940fcef0 0x7fb694071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb69b130700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 0x7fb6940702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb69b130700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb694070970 con 0x7fb6940fe0e0 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb69b130700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb694070ae0 con 0x7fb6940fcef0 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb693fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 0x7fb6940702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb693fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 0x7fb6940702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52264/0 (socket says 172.21.15.2:52264) 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.984+0000 7fb693fff700 1 -- 172.21.15.2:0/777986554 learned_addr learned my addr 172.21.15.2:0/777986554 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb698ecc700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6940fcef0 0x7fb694071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb693fff700 1 -- 172.21.15.2:0/777986554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6940fcef0 msgr2=0x7fb694071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb693fff700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6940fcef0 0x7fb694071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb693fff700 1 -- 172.21.15.2:0/777986554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb680005040 con 0x7fb6940fe0e0 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb698ecc700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6940fcef0 0x7fb694071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb693fff700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 0x7fb6940702e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb68000b3b0 tx=0x7fb680009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb6917fa700 1 -- 172.21.15.2:0/777986554 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb68000e040 con 0x7fb6940fe0e0 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb6917fa700 1 -- 172.21.15.2:0/777986554 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb680009b10 con 0x7fb6940fe0e0 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb694070d60 con 0x7fb6940fe0e0 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb6917fa700 1 -- 172.21.15.2:0/777986554 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb68001c630 con 0x7fb6940fe0e0 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.985+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6941159c0 con 0x7fb6940fe0e0 2024-02-16T22:17:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.986+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69404bed0 con 0x7fb6940fe0e0 2024-02-16T22:17:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.987+0000 7fb6917fa700 1 -- 172.21.15.2:0/777986554 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fb68000bcc0 con 0x7fb6940fe0e0 2024-02-16T22:17:35.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.987+0000 7fb6917fa700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb6840692a0 0x7fb68406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:35.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.987+0000 7fb6917fa700 1 -- 172.21.15.2:0/777986554 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb680088320 con 0x7fb6940fe0e0 2024-02-16T22:17:35.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.987+0000 7fb698ecc700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb6840692a0 0x7fb68406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:35.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.987+0000 7fb698ecc700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb6840692a0 0x7fb68406b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb688006fd0 tx=0x7fb688008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:35.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:35.991+0000 7fb6917fa700 1 -- 172.21.15.2:0/777986554 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb6800583a0 con 0x7fb6940fe0e0 2024-02-16T22:17:36.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.184+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb694115dc0 con 0x7fb6940fe0e0 2024-02-16T22:17:36.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.184+0000 7fb6917fa700 1 -- 172.21.15.2:0/777986554 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7fb680057f30 con 0x7fb6940fe0e0 2024-02-16T22:17:36.186 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:17:36.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb6840692a0 msgr2=0x7fb68406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:36.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb6840692a0 0x7fb68406b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb688006fd0 tx=0x7fb688008040 comp rx=0 tx=0).stop 2024-02-16T22:17:36.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 msgr2=0x7fb6940702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:36.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 0x7fb6940702e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb68000b3b0 tx=0x7fb680009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 shutdown_connections 2024-02-16T22:17:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb6840692a0 0x7fb68406b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6940fcef0 0x7fb694071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 --2- 172.21.15.2:0/777986554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6940fe0e0 0x7fb6940702e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 >> 172.21.15.2:0/777986554 conn(0x7fb6940f8620 msgr2=0x7fb6941013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 shutdown_connections 2024-02-16T22:17:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:36.186+0000 7fb69b130700 1 -- 172.21.15.2:0/777986554 wait complete. 2024-02-16T22:17:36.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:35 smithi103 ceph-mon[36259]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:36.963 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:36 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/777986554' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-16T22:17:36.989 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1708121852,"num_in_osds":6,"osd_in_since":1708121833,"num_remapped_pgs":0} 2024-02-16T22:17:36.990 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd dump --format=json 2024-02-16T22:17:37.254 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:37.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:36 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/777986554' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-16T22:17:37.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:37 smithi002 ceph-mon[29154]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:38.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:37 smithi103 ceph-mon[36259]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:38.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:38 smithi002 ceph-mon[29154]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:38.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:17:39.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.039+0000 7f3eb8756700 1 -- 172.21.15.2:0/2569386486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb0102f50 msgr2=0x7f3eb0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:39.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.039+0000 7f3eb8756700 1 --2- 172.21.15.2:0/2569386486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb0102f50 0x7f3eb0103320 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009ce0 tx=0x7f3ea0004d10 comp rx=0 tx=0).stop 2024-02-16T22:17:39.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.039+0000 7f3eb8756700 1 -- 172.21.15.2:0/2569386486 shutdown_connections 2024-02-16T22:17:39.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.039+0000 7f3eb8756700 1 --2- 172.21.15.2:0/2569386486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3eb01038f0 0x7f3eb00fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:39.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.039+0000 7f3eb8756700 1 --2- 172.21.15.2:0/2569386486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb0102f50 0x7f3eb0103320 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:39.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.039+0000 7f3eb8756700 1 -- 172.21.15.2:0/2569386486 >> 172.21.15.2:0/2569386486 conn(0x7f3eb00f8ae0 msgr2=0x7f3eb00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:39.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb8756700 1 -- 172.21.15.2:0/2569386486 shutdown_connections 2024-02-16T22:17:39.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb8756700 1 -- 172.21.15.2:0/2569386486 wait complete. 2024-02-16T22:17:39.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb8756700 1 Processor -- start 2024-02-16T22:17:39.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb8756700 1 -- start start 2024-02-16T22:17:39.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb8756700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3eb0102f50 0x7f3eb0113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:39.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb8756700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb01038f0 0x7f3eb0114120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:39.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb8756700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb0114770 con 0x7f3eb01038f0 2024-02-16T22:17:39.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb8756700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb010daa0 con 0x7f3eb0102f50 2024-02-16T22:17:39.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb5cf1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb01038f0 0x7f3eb0114120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:39.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb5cf1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb01038f0 0x7f3eb0114120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52292/0 (socket says 172.21.15.2:52292) 2024-02-16T22:17:39.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.040+0000 7f3eb5cf1700 1 -- 172.21.15.2:0/2284841645 learned_addr learned my addr 172.21.15.2:0/2284841645 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:17:39.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eb64f2700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3eb0102f50 0x7f3eb0113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:39.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eb5cf1700 1 -- 172.21.15.2:0/2284841645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3eb0102f50 msgr2=0x7f3eb0113be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eb5cf1700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3eb0102f50 0x7f3eb0113be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eb5cf1700 1 -- 172.21.15.2:0/2284841645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea0009940 con 0x7f3eb01038f0 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eb5cf1700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb01038f0 0x7f3eb0114120 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400c890 tx=0x7f3ea40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eaeffd700 1 -- 172.21.15.2:0/2284841645 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ea400a7d0 con 0x7f3eb01038f0 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eaeffd700 1 -- 172.21.15.2:0/2284841645 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ea400ae00 con 0x7f3eb01038f0 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb010dd80 con 0x7f3eb01038f0 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eaeffd700 1 -- 172.21.15.2:0/2284841645 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ea40113a0 con 0x7f3eb01038f0 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.041+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb010e2c0 con 0x7f3eb01038f0 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.042+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb0100260 con 0x7f3eb01038f0 2024-02-16T22:17:39.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.043+0000 7f3eaeffd700 1 -- 172.21.15.2:0/2284841645 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3ea4012400 con 0x7f3eb01038f0 2024-02-16T22:17:39.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.043+0000 7f3eaeffd700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e980692a0 0x7f3e9806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:39.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.043+0000 7f3eaeffd700 1 -- 172.21.15.2:0/2284841645 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3ea4086e80 con 0x7f3eb01038f0 2024-02-16T22:17:39.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.043+0000 7f3eb64f2700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e980692a0 0x7f3e9806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:39.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.043+0000 7f3eb64f2700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e980692a0 0x7f3e9806b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000b740 tx=0x7f3ea0004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:39.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.047+0000 7f3eaeffd700 1 -- 172.21.15.2:0/2284841645 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3ea4057020 con 0x7f3eb01038f0 2024-02-16T22:17:39.227 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:38 smithi103 ceph-mon[36259]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:39.227 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:17:39.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.237+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3eb00fc6e0 con 0x7f3eb01038f0 2024-02-16T22:17:39.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.238+0000 7f3eaeffd700 1 -- 172.21.15.2:0/2284841645 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11003 (secure 0 0 0) 0x7f3ea4051e60 con 0x7f3eb01038f0 2024-02-16T22:17:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:17:39.240 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":40,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","created":"2024-02-16T22:11:59.286486+0000","modified":"2024-02-16T22:17:33.876912+0000","last_up_change":"2024-02-16T22:17:32.871791+0000","last_in_change":"2024-02-16T22:17:13.716253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-16T22:16:18.867363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"348575cf-dd0d-444c-bab9-bd51dcc9189c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6803","nonce":3509867548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6805","nonce":3509867548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6809","nonce":3509867548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6807","nonce":3509867548}]},"public_addr":"172.21.15.2:6803/3509867548","cluster_addr":"172.21.15.2:6805/3509867548","heartbeat_back_addr":"172.21.15.2:6809/3509867548","heartbeat_front_addr":"172.21.15.2:6807/3509867548","state":["exists","up"]},{"osd":1,"uuid":"eaffda16-c0db-440f-9a1a-99e228b3dc19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6811","nonce":609822393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6813","nonce":609822393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6817","nonce":609822393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6815","nonce":609822393}]},"public_addr":"172.21.15.2:6811/609822393","cluster_addr":"172.21.15.2:6813/609822393","heartbeat_back_addr":"172.21.15.2:6817/609822393","heartbeat_front_addr":"172.21.15.2:6815/609822393","state":["exists","up"]},{"osd":2,"uuid":"732c0a64-5183-42aa-8220-a12116e52442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6819","nonce":2737615042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6821","nonce":2737615042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6825","nonce":2737615042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6823","nonce":2737615042}]},"public_addr":"172.21.15.2:6819/2737615042","cluster_addr":"172.21.15.2:6821/2737615042","heartbeat_back_addr":"172.21.15.2:6825/2737615042","heartbeat_front_addr":"172.21.15.2:6823/2737615042","state":["exists","up"]},{"osd":3,"uuid":"09a182db-e441-45f8-9d13-61d69b7db54d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6801","nonce":425086385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6803","nonce":425086385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6807","nonce":425086385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6805","nonce":425086385}]},"public_addr":"172.21.15.103:6801/425086385","cluster_addr":"172.21.15.103:6803/425086385","heartbeat_back_addr":"172.21.15.103:6807/425086385","heartbeat_front_addr":"172.21.15.103:6805/425086385","state":["exists","up"]},{"osd":4,"uuid":"ee9a7426-801d-420f-be92-cd18c21d11b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6809","nonce":420978374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6811","nonce":420978374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6815","nonce":420978374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6813","nonce":420978374}]},"public_addr":"172.21.15.103:6809/420978374","cluster_addr":"172.21.15.103:6811/420978374","heartbeat_back_addr":"172.21.15.103:6815/420978374","heartbeat_front_addr":"172.21.15.103:6813/420978374","state":["exists","up"]},{"osd":5,"uuid":"68c72748-2d27-4ada-a35b-53e1bab20f9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6817","nonce":3241271941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6819","nonce":3241271941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6823","nonce":3241271941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6821","nonce":3241271941}]},"public_addr":"172.21.15.103:6817/3241271941","cluster_addr":"172.21.15.103:6819/3241271941","heartbeat_back_addr":"172.21.15.103:6823/3241271941","heartbeat_front_addr":"172.21.15.103:6821/3241271941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:15:21.168791+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:15:48.309630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:16:16.840477+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:16:40.949660+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:17:05.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:17:30.935853+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/1058268540":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6801/2481260931":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6800/2481260931":"2024-02-17T22:14:08.529242+0000","172.21.15.2:0/504770625":"2024-02-17T22:14:08.529242+0000","172.21.15.2:0/1364558863":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6801/673699505":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/135684194":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/3921669523":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/1318894983":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/3292480868":"2024-02-17T22:13:00.117629+0000","172.21.15.2:6801/395571706":"2024-02-17T22:12:27.545802+0000","172.21.15.2:6800/673699505":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/3834071444":"2024-02-17T22:12:27.545802+0000","172.21.15.2:6800/395571706":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/495154782":"2024-02-17T22:13:00.117629+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e980692a0 msgr2=0x7f3e9806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e980692a0 0x7f3e9806b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000b740 tx=0x7f3ea0004d10 comp rx=0 tx=0).stop 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb01038f0 msgr2=0x7f3eb0114120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb01038f0 0x7f3eb0114120 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400c890 tx=0x7f3ea40044c0 comp rx=0 tx=0).stop 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 shutdown_connections 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e980692a0 0x7f3e9806b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3eb0102f50 0x7f3eb0113be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 --2- 172.21.15.2:0/2284841645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3eb01038f0 0x7f3eb0114120 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 >> 172.21.15.2:0/2284841645 conn(0x7f3eb00f8ae0 msgr2=0x7f3eb00fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 shutdown_connections 2024-02-16T22:17:39.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:39.240+0000 7f3eb8756700 1 -- 172.21.15.2:0/2284841645 wait complete. 2024-02-16T22:17:39.981 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-16T22:16:18.867363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-16T22:17:39.981 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-16T22:17:40.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:39 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2284841645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-16T22:17:40.246 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:40.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:39 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2284841645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-16T22:17:40.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:40 smithi002 ceph-mon[29154]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:41.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:40 smithi103 ceph-mon[36259]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:42.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.109+0000 7f6c4af9c700 1 -- 172.21.15.2:0/2151582223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c440fb6c0 msgr2=0x7f6c440fbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:42.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.109+0000 7f6c4af9c700 1 --2- 172.21.15.2:0/2151582223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c440fb6c0 0x7f6c440fbb30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6c30004a50 tx=0x7f6c3000af40 comp rx=0 tx=0).stop 2024-02-16T22:17:42.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.109+0000 7f6c4af9c700 1 -- 172.21.15.2:0/2151582223 shutdown_connections 2024-02-16T22:17:42.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.109+0000 7f6c4af9c700 1 --2- 172.21.15.2:0/2151582223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c440fb6c0 0x7f6c440fbb30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:42.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.109+0000 7f6c4af9c700 1 --2- 172.21.15.2:0/2151582223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c441016b0 0x7f6c44101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:42.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.109+0000 7f6c4af9c700 1 -- 172.21.15.2:0/2151582223 >> 172.21.15.2:0/2151582223 conn(0x7f6c44071560 msgr2=0x7f6c44071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:42.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.109+0000 7f6c4af9c700 1 -- 172.21.15.2:0/2151582223 shutdown_connections 2024-02-16T22:17:42.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.109+0000 7f6c4af9c700 1 -- 172.21.15.2:0/2151582223 wait complete. 2024-02-16T22:17:42.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c4af9c700 1 Processor -- start 2024-02-16T22:17:42.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c4af9c700 1 -- start start 2024-02-16T22:17:42.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c4af9c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440fb6c0 0x7f6c44113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c4af9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c441016b0 0x7f6c44113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c4af9c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c441145b0 con 0x7f6c441016b0 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c4af9c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4410d8e0 con 0x7f6c440fb6c0 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c43fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c441016b0 0x7f6c44113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c43fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c441016b0 0x7f6c44113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42672/0 (socket says 172.21.15.2:42672) 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c43fff700 1 -- 172.21.15.2:0/3900346561 learned_addr learned my addr 172.21.15.2:0/3900346561 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.110+0000 7f6c48d38700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440fb6c0 0x7f6c44113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.111+0000 7f6c43fff700 1 -- 172.21.15.2:0/3900346561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440fb6c0 msgr2=0x7f6c44113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.111+0000 7f6c43fff700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440fb6c0 0x7f6c44113a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.111+0000 7f6c43fff700 1 -- 172.21.15.2:0/3900346561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c30005040 con 0x7f6c441016b0 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.111+0000 7f6c43fff700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c441016b0 0x7f6c44113f60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6c30004710 tx=0x7f6c3000af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.111+0000 7f6c417fa700 1 -- 172.21.15.2:0/3900346561 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c3000f070 con 0x7f6c441016b0 2024-02-16T22:17:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.111+0000 7f6c417fa700 1 -- 172.21.15.2:0/3900346561 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c3000a4c0 con 0x7f6c441016b0 2024-02-16T22:17:42.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.111+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4410db60 con 0x7f6c441016b0 2024-02-16T22:17:42.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.111+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4410e040 con 0x7f6c441016b0 2024-02-16T22:17:42.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.111+0000 7f6c417fa700 1 -- 172.21.15.2:0/3900346561 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c30014800 con 0x7f6c441016b0 2024-02-16T22:17:42.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.112+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c44103b30 con 0x7f6c441016b0 2024-02-16T22:17:42.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.112+0000 7f6c417fa700 1 -- 172.21.15.2:0/3900346561 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f6c30014a70 con 0x7f6c441016b0 2024-02-16T22:17:42.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.113+0000 7f6c417fa700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6c340692a0 0x7f6c3406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:42.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.113+0000 7f6c417fa700 1 -- 172.21.15.2:0/3900346561 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6c30088790 con 0x7f6c441016b0 2024-02-16T22:17:42.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.113+0000 7f6c48d38700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6c340692a0 0x7f6c3406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:42.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.113+0000 7f6c48d38700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6c340692a0 0x7f6c3406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6c38004f40 tx=0x7f6c3800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:42.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.117+0000 7f6c417fa700 1 -- 172.21.15.2:0/3900346561 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6c30058890 con 0x7f6c441016b0 2024-02-16T22:17:42.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.311+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c4410f000 con 0x7f6c441016b0 2024-02-16T22:17:42.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.312+0000 7f6c417fa700 1 -- 172.21.15.2:0/3900346561 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c30058420 con 0x7f6c441016b0 2024-02-16T22:17:42.314 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-02-16T22:17:42.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6c340692a0 msgr2=0x7f6c3406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:42.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6c340692a0 0x7f6c3406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6c38004f40 tx=0x7f6c3800b040 comp rx=0 tx=0).stop 2024-02-16T22:17:42.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c441016b0 msgr2=0x7f6c44113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:42.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c441016b0 0x7f6c44113f60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6c30004710 tx=0x7f6c3000af40 comp rx=0 tx=0).stop 2024-02-16T22:17:42.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 shutdown_connections 2024-02-16T22:17:42.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f6c340692a0 0x7f6c3406b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:42.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c440fb6c0 0x7f6c44113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:42.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 --2- 172.21.15.2:0/3900346561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c441016b0 0x7f6c44113f60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:42.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 >> 172.21.15.2:0/3900346561 conn(0x7f6c44071560 msgr2=0x7f6c440fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:42.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 shutdown_connections 2024-02-16T22:17:42.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:42.314+0000 7f6c4af9c700 1 -- 172.21.15.2:0/3900346561 wait complete. 2024-02-16T22:17:42.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:42 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3900346561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-16T22:17:42.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:42 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3900346561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-16T22:17:43.127 INFO:tasks.cephadm:Setting up client nodes... 2024-02-16T22:17:43.127 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-16T22:17:43.390 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:43.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:43 smithi002 ceph-mon[29154]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:43.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:43 smithi103 ceph-mon[36259]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.269+0000 7f2668eb2700 1 -- 172.21.15.2:0/427252127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fbe40 msgr2=0x7f26640fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.269+0000 7f2668eb2700 1 --2- 172.21.15.2:0/427252127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fbe40 0x7f26640fc250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2654004940 tx=0x7f2654009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.269+0000 7f2668eb2700 1 -- 172.21.15.2:0/427252127 shutdown_connections 2024-02-16T22:17:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.269+0000 7f2668eb2700 1 --2- 172.21.15.2:0/427252127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26640fcfb0 0x7f26640fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.269+0000 7f2668eb2700 1 --2- 172.21.15.2:0/427252127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fbe40 0x7f26640fc250 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.269+0000 7f2668eb2700 1 -- 172.21.15.2:0/427252127 >> 172.21.15.2:0/427252127 conn(0x7f26640f74e0 msgr2=0x7f26640f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.269+0000 7f2668eb2700 1 -- 172.21.15.2:0/427252127 shutdown_connections 2024-02-16T22:17:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.269+0000 7f2668eb2700 1 -- 172.21.15.2:0/427252127 wait complete. 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f2668eb2700 1 Processor -- start 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f2668eb2700 1 -- start start 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f2668eb2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26640fbe40 0x7f266410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f2668eb2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fcfb0 0x7f266410de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f2668eb2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266410e500 con 0x7f26640fcfb0 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f2668eb2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266410ed40 con 0x7f26640fbe40 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f2661d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fcfb0 0x7f266410de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f2661d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fcfb0 0x7f266410de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42692/0 (socket says 172.21.15.2:42692) 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f2661d9b700 1 -- 172.21.15.2:0/2844863801 learned_addr learned my addr 172.21.15.2:0/2844863801 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.270+0000 7f266259c700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26640fbe40 0x7f266410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:45.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.271+0000 7f2661d9b700 1 -- 172.21.15.2:0/2844863801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26640fbe40 msgr2=0x7f266410d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.271+0000 7f2661d9b700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26640fbe40 0x7f266410d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.271+0000 7f2661d9b700 1 -- 172.21.15.2:0/2844863801 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654005040 con 0x7f26640fcfb0 2024-02-16T22:17:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.271+0000 7f2661d9b700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fcfb0 0x7f266410de70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f265800b790 tx=0x7f2658009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.271+0000 7f2652ffd700 1 -- 172.21.15.2:0/2844863801 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f265800f6f0 con 0x7f26640fcfb0 2024-02-16T22:17:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.271+0000 7f2652ffd700 1 -- 172.21.15.2:0/2844863801 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f265800fd20 con 0x7f26640fcfb0 2024-02-16T22:17:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.271+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266410f020 con 0x7f26640fcfb0 2024-02-16T22:17:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.271+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266410f560 con 0x7f26640fcfb0 2024-02-16T22:17:45.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.271+0000 7f2652ffd700 1 -- 172.21.15.2:0/2844863801 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26580103a0 con 0x7f26640fcfb0 2024-02-16T22:17:45.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.272+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26440052e0 con 0x7f26640fcfb0 2024-02-16T22:17:45.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.273+0000 7f2652ffd700 1 -- 172.21.15.2:0/2844863801 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2658011400 con 0x7f26640fcfb0 2024-02-16T22:17:45.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.273+0000 7f2652ffd700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f264c0692a0 0x7f264c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:45.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.273+0000 7f2652ffd700 1 -- 172.21.15.2:0/2844863801 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2658086c00 con 0x7f26640fcfb0 2024-02-16T22:17:45.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.273+0000 7f266259c700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f264c0692a0 0x7f264c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:45.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.273+0000 7f266259c700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f264c0692a0 0x7f264c06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2654004f90 tx=0x7f2654009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:45.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.278+0000 7f2652ffd700 1 -- 172.21.15.2:0/2844863801 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f265808c020 con 0x7f26640fcfb0 2024-02-16T22:17:45.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.544+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f2644005590 con 0x7f26640fcfb0 2024-02-16T22:17:45.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.548+0000 7f2652ffd700 1 -- 172.21.15.2:0/2844863801 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f2658056a70 con 0x7f26640fcfb0 2024-02-16T22:17:45.550 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-02-16T22:17:45.550 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQAJ389lnw6LIBAAc/IQG/mTSuhtoDBHoz7lIw== 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f264c0692a0 msgr2=0x7f264c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f264c0692a0 0x7f264c06b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2654004f90 tx=0x7f2654009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fcfb0 msgr2=0x7f266410de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fcfb0 0x7f266410de70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f265800b790 tx=0x7f2658009250 comp rx=0 tx=0).stop 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 shutdown_connections 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f264c0692a0 0x7f264c06b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26640fbe40 0x7f266410d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 --2- 172.21.15.2:0/2844863801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26640fcfb0 0x7f266410de70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 >> 172.21.15.2:0/2844863801 conn(0x7f26640f74e0 msgr2=0x7f26640656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 shutdown_connections 2024-02-16T22:17:45.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:45.550+0000 7f2668eb2700 1 -- 172.21.15.2:0/2844863801 wait complete. 2024-02-16T22:17:46.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:45 smithi002 ceph-mon[29154]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:46.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:45 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2844863801' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-16T22:17:46.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:45 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2844863801' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-16T22:17:46.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:45 smithi103 ceph-mon[36259]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:46.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:45 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2844863801' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-16T22:17:46.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:45 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2844863801' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-16T22:17:46.345 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:17:46.345 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-16T22:17:46.345 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-16T22:17:46.391 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-16T22:17:46.656 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi103/config 2024-02-16T22:17:47.910 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:47 smithi103 ceph-mon[36259]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.143+0000 7fdd85506700 1 -- 172.21.15.103:0/1171275274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 msgr2=0x7fdd800fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.143+0000 7fdd85506700 1 --2- 172.21.15.103:0/1171275274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 0x7fdd800fb630 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009ce0 tx=0x7fdd70004d10 comp rx=0 tx=0).stop 2024-02-16T22:17:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.143+0000 7fdd85506700 1 -- 172.21.15.103:0/1171275274 shutdown_connections 2024-02-16T22:17:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.143+0000 7fdd85506700 1 --2- 172.21.15.103:0/1171275274 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd800fe2b0 0x7fdd800fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.143+0000 7fdd85506700 1 --2- 172.21.15.103:0/1171275274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 0x7fdd800fb630 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.143+0000 7fdd85506700 1 -- 172.21.15.103:0/1171275274 >> 172.21.15.103:0/1171275274 conn(0x7fdd800f6ca0 msgr2=0x7fdd800f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:48.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.143+0000 7fdd85506700 1 -- 172.21.15.103:0/1171275274 shutdown_connections 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.143+0000 7fdd85506700 1 -- 172.21.15.103:0/1171275274 wait complete. 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd85506700 1 Processor -- start 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd85506700 1 -- start start 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd85506700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 0x7fdd8010db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd85506700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd800fe2b0 0x7fdd8010e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd85506700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8010e760 con 0x7fdd800fe2b0 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd85506700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8010efa0 con 0x7fdd800fb220 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd7effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 0x7fdd8010db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd7effd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 0x7fdd8010db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43520/0 (socket says 172.21.15.103:43520) 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd7effd700 1 -- 172.21.15.103:0/815742813 learned_addr learned my addr 172.21.15.103:0/815742813 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd7e7fc700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd800fe2b0 0x7fdd8010e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd7effd700 1 -- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd800fe2b0 msgr2=0x7fdd8010e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd7effd700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd800fe2b0 0x7fdd8010e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd7effd700 1 -- 172.21.15.103:0/815742813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd70009940 con 0x7fdd800fb220 2024-02-16T22:17:48.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd7effd700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 0x7fdd8010db90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000bf90 tx=0x7fdd70004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:48.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd777fe700 1 -- 172.21.15.103:0/815742813 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd7000e040 con 0x7fdd800fb220 2024-02-16T22:17:48.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.144+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8010f220 con 0x7fdd800fb220 2024-02-16T22:17:48.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.145+0000 7fdd777fe700 1 -- 172.21.15.103:0/815742813 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd700128e0 con 0x7fdd800fb220 2024-02-16T22:17:48.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.145+0000 7fdd777fe700 1 -- 172.21.15.103:0/815742813 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd7001c990 con 0x7fdd800fb220 2024-02-16T22:17:48.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.145+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8010f760 con 0x7fdd800fb220 2024-02-16T22:17:48.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.146+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8004bed0 con 0x7fdd800fb220 2024-02-16T22:17:48.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.146+0000 7fdd777fe700 1 -- 172.21.15.103:0/815742813 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fdd70012410 con 0x7fdd800fb220 2024-02-16T22:17:48.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.146+0000 7fdd777fe700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdd6c0692a0 0x7fdd6c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:48.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.146+0000 7fdd777fe700 1 -- 172.21.15.103:0/815742813 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdd70088b10 con 0x7fdd800fb220 2024-02-16T22:17:48.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.146+0000 7fdd7e7fc700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdd6c0692a0 0x7fdd6c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:48.149 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.147+0000 7fdd7e7fc700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdd6c0692a0 0x7fdd6c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdd68006fd0 tx=0x7fdd68008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:48.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.151+0000 7fdd777fe700 1 -- 172.21.15.103:0/815742813 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdd70058c10 con 0x7fdd800fb220 2024-02-16T22:17:48.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:47 smithi002 ceph-mon[29154]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:48.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.408+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fdd8010fb60 con 0x7fdd800fb220 2024-02-16T22:17:48.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.413+0000 7fdd777fe700 1 -- 172.21.15.103:0/815742813 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fdd700587a0 con 0x7fdd800fb220 2024-02-16T22:17:48.415 INFO:teuthology.orchestra.run.smithi103.stdout:[client.1] 2024-02-16T22:17:48.415 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQAM389lYSZxGBAA5yO86jgX/0+16dYNSloVvA== 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdd6c0692a0 msgr2=0x7fdd6c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdd6c0692a0 0x7fdd6c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdd68006fd0 tx=0x7fdd68008040 comp rx=0 tx=0).stop 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 msgr2=0x7fdd8010db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 0x7fdd8010db90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000bf90 tx=0x7fdd70004d10 comp rx=0 tx=0).stop 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 shutdown_connections 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdd6c0692a0 0x7fdd6c06b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd800fb220 0x7fdd8010db90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 --2- 172.21.15.103:0/815742813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd800fe2b0 0x7fdd8010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 >> 172.21.15.103:0/815742813 conn(0x7fdd800f6ca0 msgr2=0x7fdd801014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 shutdown_connections 2024-02-16T22:17:48.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-02-16T22:17:48.415+0000 7fdd85506700 1 -- 172.21.15.103:0/815742813 wait complete. 2024-02-16T22:17:48.984 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:17:48.984 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-16T22:17:48.984 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-16T22:17:49.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:48 smithi103 ceph-mon[36259]: from='client.? 172.21.15.103:0/815742813' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-16T22:17:49.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:48 smithi103 ceph-mon[36259]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-16T22:17:49.026 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:48 smithi103 ceph-mon[36259]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-16T22:17:49.028 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-16T22:17:49.028 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-16T22:17:49.028 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph mgr dump --format=json 2024-02-16T22:17:49.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:48 smithi002 ceph-mon[29154]: from='client.? 172.21.15.103:0/815742813' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-16T22:17:49.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:48 smithi002 ceph-mon[29154]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-16T22:17:49.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:48 smithi002 ceph-mon[29154]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-16T22:17:49.297 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:49.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:49 smithi002 ceph-mon[29154]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:49 smithi103 ceph-mon[36259]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:51.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 -- 172.21.15.2:0/3035325502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 msgr2=0x7ff698103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:51.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 --2- 172.21.15.2:0/3035325502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 0x7ff698103320 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff690009ce0 tx=0x7ff690004d10 comp rx=0 tx=0).stop 2024-02-16T22:17:51.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 -- 172.21.15.2:0/3035325502 shutdown_connections 2024-02-16T22:17:51.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 --2- 172.21.15.2:0/3035325502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6981038f0 0x7ff6980fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:51.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 --2- 172.21.15.2:0/3035325502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 0x7ff698103320 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:51.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 -- 172.21.15.2:0/3035325502 >> 172.21.15.2:0/3035325502 conn(0x7ff6980f8ae0 msgr2=0x7ff6980faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:51.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 -- 172.21.15.2:0/3035325502 shutdown_connections 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 -- 172.21.15.2:0/3035325502 wait complete. 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 Processor -- start 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.024+0000 7ff69c986700 1 -- start start 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69c986700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 0x7ff698113c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69c986700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6981038f0 0x7ff698114160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69c986700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff698114840 con 0x7ff698102f50 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69c986700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69810dae0 con 0x7ff6981038f0 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 0x7ff698113c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 0x7ff698113c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46750/0 (socket says 172.21.15.2:46750) 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69659c700 1 -- 172.21.15.2:0/1848271316 learned_addr learned my addr 172.21.15.2:0/1848271316 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff695d9b700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6981038f0 0x7ff698114160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69659c700 1 -- 172.21.15.2:0/1848271316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6981038f0 msgr2=0x7ff698114160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:51.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69659c700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6981038f0 0x7ff698114160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:51.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69659c700 1 -- 172.21.15.2:0/1848271316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff690009940 con 0x7ff698102f50 2024-02-16T22:17:51.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69659c700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 0x7ff698113c20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff690004b70 tx=0x7ff690004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:51.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff68effd700 1 -- 172.21.15.2:0/1848271316 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff69000e040 con 0x7ff698102f50 2024-02-16T22:17:51.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69810dd60 con 0x7ff698102f50 2024-02-16T22:17:51.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff68effd700 1 -- 172.21.15.2:0/1848271316 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff690004610 con 0x7ff698102f50 2024-02-16T22:17:51.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff68effd700 1 -- 172.21.15.2:0/1848271316 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff69001c810 con 0x7ff698102f50 2024-02-16T22:17:51.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.025+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69810e2a0 con 0x7ff698102f50 2024-02-16T22:17:51.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.026+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6980fc6e0 con 0x7ff698102f50 2024-02-16T22:17:51.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.027+0000 7ff68effd700 1 -- 172.21.15.2:0/1848271316 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff690012c00 con 0x7ff698102f50 2024-02-16T22:17:51.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.027+0000 7ff68effd700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6840692b0 0x7ff68406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:51.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.027+0000 7ff68effd700 1 -- 172.21.15.2:0/1848271316 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff690088ac0 con 0x7ff698102f50 2024-02-16T22:17:51.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.027+0000 7ff695d9b700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6840692b0 0x7ff68406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:51.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.028+0000 7ff695d9b700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6840692b0 0x7ff68406b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff680006fd0 tx=0x7ff680008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:51.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.032+0000 7ff68effd700 1 -- 172.21.15.2:0/1848271316 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff690058b40 con 0x7ff698102f50 2024-02-16T22:17:51.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.276+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff69810f200 con 0x7ff698102f50 2024-02-16T22:17:51.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.279+0000 7ff68effd700 1 -- 172.21.15.2:0/1848271316 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167162 (secure 0 0 0) 0x7ff6900586d0 con 0x7ff698102f50 2024-02-16T22:17:51.283 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:17:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.282+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6840692b0 msgr2=0x7ff68406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.282+0000 7ff69c986700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6840692b0 0x7ff68406b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff680006fd0 tx=0x7ff680008040 comp rx=0 tx=0).stop 2024-02-16T22:17:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.282+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 msgr2=0x7ff698113c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.282+0000 7ff69c986700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 0x7ff698113c20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff690004b70 tx=0x7ff690004d10 comp rx=0 tx=0).stop 2024-02-16T22:17:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.282+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 shutdown_connections 2024-02-16T22:17:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.282+0000 7ff69c986700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6840692b0 0x7ff68406b760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.282+0000 7ff69c986700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff698102f50 0x7ff698113c20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.282+0000 7ff69c986700 1 --2- 172.21.15.2:0/1848271316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6981038f0 0x7ff698114160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:51.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.282+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 >> 172.21.15.2:0/1848271316 conn(0x7ff6980f8ae0 msgr2=0x7ff6980fa4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:51.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.283+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 shutdown_connections 2024-02-16T22:17:51.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:51.283+0000 7ff69c986700 1 -- 172.21.15.2:0/1848271316 wait complete. 2024-02-16T22:17:51.988 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":19,"active_gid":14212,"active_name":"smithi002.resmjv","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":1176790314},{"type":"v1","addr":"172.21.15.2:6801","nonce":1176790314}]},"active_addr":"172.21.15.2:6801/1176790314","active_change":"2024-02-16T22:14:08.529359+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14232,"name":"smithi103.ydydbo","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.2:8443/","prometheus":"http://172.21.15.2:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2359240530}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3914613851}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1593731102}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1759557733}]}]} 2024-02-16T22:17:51.992 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-16T22:17:51.992 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-16T22:17:51.992 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd dump --format=json 2024-02-16T22:17:52.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:51 smithi002 ceph-mon[29154]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:52.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:51 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1848271316' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-16T22:17:52.253 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:52.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:51 smithi103 ceph-mon[36259]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:52.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:51 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1848271316' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-16T22:17:53.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.895+0000 7fdc7497a700 1 -- 172.21.15.2:0/2352627500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 msgr2=0x7fdc70102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.895+0000 7fdc7497a700 1 --2- 172.21.15.2:0/2352627500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 0x7fdc70102210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdc60004940 tx=0x7fdc60009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.895+0000 7fdc7497a700 1 -- 172.21.15.2:0/2352627500 shutdown_connections 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.895+0000 7fdc7497a700 1 --2- 172.21.15.2:0/2352627500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc701027e0 0x7fdc700fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.895+0000 7fdc7497a700 1 --2- 172.21.15.2:0/2352627500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 0x7fdc70102210 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.895+0000 7fdc7497a700 1 -- 172.21.15.2:0/2352627500 >> 172.21.15.2:0/2352627500 conn(0x7fdc70071d90 msgr2=0x7fdc70072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.895+0000 7fdc7497a700 1 -- 172.21.15.2:0/2352627500 shutdown_connections 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.895+0000 7fdc7497a700 1 -- 172.21.15.2:0/2352627500 wait complete. 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.895+0000 7fdc7497a700 1 Processor -- start 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc7497a700 1 -- start start 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc7497a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 0x7fdc70112970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc7497a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc701027e0 0x7fdc70112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc7497a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70113590 con 0x7fdc70101e40 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc7497a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7010c830 con 0x7fdc701027e0 2024-02-16T22:17:53.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc6e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 0x7fdc70112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc6e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 0x7fdc70112970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46774/0 (socket says 172.21.15.2:46774) 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc6e59c700 1 -- 172.21.15.2:0/3358494837 learned_addr learned my addr 172.21.15.2:0/3358494837 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc6dd9b700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc701027e0 0x7fdc70112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc6e59c700 1 -- 172.21.15.2:0/3358494837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc701027e0 msgr2=0x7fdc70112eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc6e59c700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc701027e0 0x7fdc70112eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc6e59c700 1 -- 172.21.15.2:0/3358494837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc60005040 con 0x7fdc70101e40 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc6e59c700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 0x7fdc70112970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009fd0 tx=0x7fdc60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.896+0000 7fdc5effd700 1 -- 172.21.15.2:0/3358494837 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc6000e040 con 0x7fdc70101e40 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.897+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7010cab0 con 0x7fdc70101e40 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.897+0000 7fdc5effd700 1 -- 172.21.15.2:0/3358494837 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc6000bbf0 con 0x7fdc70101e40 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.897+0000 7fdc5effd700 1 -- 172.21.15.2:0/3358494837 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc6001c3f0 con 0x7fdc70101e40 2024-02-16T22:17:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.897+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7010cff0 con 0x7fdc70101e40 2024-02-16T22:17:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.898+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc700738b0 con 0x7fdc70101e40 2024-02-16T22:17:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.898+0000 7fdc5effd700 1 -- 172.21.15.2:0/3358494837 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fdc6000bd60 con 0x7fdc70101e40 2024-02-16T22:17:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.899+0000 7fdc5effd700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdc580692b0 0x7fdc5806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.899+0000 7fdc6dd9b700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdc580692b0 0x7fdc5806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.899+0000 7fdc5effd700 1 -- 172.21.15.2:0/3358494837 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdc60088190 con 0x7fdc70101e40 2024-02-16T22:17:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.899+0000 7fdc6dd9b700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdc580692b0 0x7fdc5806b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdc64006fd0 tx=0x7fdc64008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:53.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:53.903+0000 7fdc5effd700 1 -- 172.21.15.2:0/3358494837 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdc60058290 con 0x7fdc70101e40 2024-02-16T22:17:53.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:53 smithi002 ceph-mon[29154]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:53.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:17:54.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.094+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc7010d210 con 0x7fdc70101e40 2024-02-16T22:17:54.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.094+0000 7fdc5effd700 1 -- 172.21.15.2:0/3358494837 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11003 (secure 0 0 0) 0x7fdc60057e20 con 0x7fdc70101e40 2024-02-16T22:17:54.096 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:17:54.096 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":40,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","created":"2024-02-16T22:11:59.286486+0000","modified":"2024-02-16T22:17:33.876912+0000","last_up_change":"2024-02-16T22:17:32.871791+0000","last_in_change":"2024-02-16T22:17:13.716253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-16T22:16:18.867363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"348575cf-dd0d-444c-bab9-bd51dcc9189c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6803","nonce":3509867548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6805","nonce":3509867548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6809","nonce":3509867548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6807","nonce":3509867548}]},"public_addr":"172.21.15.2:6803/3509867548","cluster_addr":"172.21.15.2:6805/3509867548","heartbeat_back_addr":"172.21.15.2:6809/3509867548","heartbeat_front_addr":"172.21.15.2:6807/3509867548","state":["exists","up"]},{"osd":1,"uuid":"eaffda16-c0db-440f-9a1a-99e228b3dc19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6811","nonce":609822393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6813","nonce":609822393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6817","nonce":609822393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6815","nonce":609822393}]},"public_addr":"172.21.15.2:6811/609822393","cluster_addr":"172.21.15.2:6813/609822393","heartbeat_back_addr":"172.21.15.2:6817/609822393","heartbeat_front_addr":"172.21.15.2:6815/609822393","state":["exists","up"]},{"osd":2,"uuid":"732c0a64-5183-42aa-8220-a12116e52442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6819","nonce":2737615042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6821","nonce":2737615042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6825","nonce":2737615042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6823","nonce":2737615042}]},"public_addr":"172.21.15.2:6819/2737615042","cluster_addr":"172.21.15.2:6821/2737615042","heartbeat_back_addr":"172.21.15.2:6825/2737615042","heartbeat_front_addr":"172.21.15.2:6823/2737615042","state":["exists","up"]},{"osd":3,"uuid":"09a182db-e441-45f8-9d13-61d69b7db54d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6801","nonce":425086385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6803","nonce":425086385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6807","nonce":425086385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6805","nonce":425086385}]},"public_addr":"172.21.15.103:6801/425086385","cluster_addr":"172.21.15.103:6803/425086385","heartbeat_back_addr":"172.21.15.103:6807/425086385","heartbeat_front_addr":"172.21.15.103:6805/425086385","state":["exists","up"]},{"osd":4,"uuid":"ee9a7426-801d-420f-be92-cd18c21d11b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6809","nonce":420978374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6811","nonce":420978374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6815","nonce":420978374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6813","nonce":420978374}]},"public_addr":"172.21.15.103:6809/420978374","cluster_addr":"172.21.15.103:6811/420978374","heartbeat_back_addr":"172.21.15.103:6815/420978374","heartbeat_front_addr":"172.21.15.103:6813/420978374","state":["exists","up"]},{"osd":5,"uuid":"68c72748-2d27-4ada-a35b-53e1bab20f9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6817","nonce":3241271941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6819","nonce":3241271941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6823","nonce":3241271941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6821","nonce":3241271941}]},"public_addr":"172.21.15.103:6817/3241271941","cluster_addr":"172.21.15.103:6819/3241271941","heartbeat_back_addr":"172.21.15.103:6823/3241271941","heartbeat_front_addr":"172.21.15.103:6821/3241271941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:15:21.168791+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:15:48.309630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:16:16.840477+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:16:40.949660+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:17:05.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:17:30.935853+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/1058268540":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6801/2481260931":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6800/2481260931":"2024-02-17T22:14:08.529242+0000","172.21.15.2:0/504770625":"2024-02-17T22:14:08.529242+0000","172.21.15.2:0/1364558863":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6801/673699505":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/135684194":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/3921669523":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/1318894983":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/3292480868":"2024-02-17T22:13:00.117629+0000","172.21.15.2:6801/395571706":"2024-02-17T22:12:27.545802+0000","172.21.15.2:6800/673699505":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/3834071444":"2024-02-17T22:12:27.545802+0000","172.21.15.2:6800/395571706":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/495154782":"2024-02-17T22:13:00.117629+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-16T22:17:54.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdc580692b0 msgr2=0x7fdc5806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:54.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdc580692b0 0x7fdc5806b760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdc64006fd0 tx=0x7fdc64008040 comp rx=0 tx=0).stop 2024-02-16T22:17:54.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 msgr2=0x7fdc70112970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:54.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 0x7fdc70112970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009fd0 tx=0x7fdc60009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:54.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 shutdown_connections 2024-02-16T22:17:54.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdc580692b0 0x7fdc5806b760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:54.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc70101e40 0x7fdc70112970 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:54.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 --2- 172.21.15.2:0/3358494837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc701027e0 0x7fdc70112eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:54.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 >> 172.21.15.2:0/3358494837 conn(0x7fdc70071d90 msgr2=0x7fdc700fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:54.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 shutdown_connections 2024-02-16T22:17:54.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:54.096+0000 7fdc7497a700 1 -- 172.21.15.2:0/3358494837 wait complete. 2024-02-16T22:17:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:53 smithi103 ceph-mon[36259]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:17:54.860 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-16T22:17:54.861 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd dump --format=json 2024-02-16T22:17:55.120 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:54 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3358494837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-16T22:17:55.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:54 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3358494837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-16T22:17:56.166 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:55 smithi002 ceph-mon[29154]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:56.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:55 smithi103 ceph-mon[36259]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:56.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.658+0000 7f216cc17700 1 -- 172.21.15.2:0/256637472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21681027f0 msgr2=0x7f21680fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:56.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.658+0000 7f216cc17700 1 --2- 172.21.15.2:0/256637472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21681027f0 0x7f21680fbe60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2150004940 tx=0x7f2150009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:56.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.658+0000 7f216cc17700 1 -- 172.21.15.2:0/256637472 shutdown_connections 2024-02-16T22:17:56.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.658+0000 7f216cc17700 1 --2- 172.21.15.2:0/256637472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21681027f0 0x7f21680fbe60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:56.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.658+0000 7f216cc17700 1 --2- 172.21.15.2:0/256637472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2168101e50 0x7f2168102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:56.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.658+0000 7f216cc17700 1 -- 172.21.15.2:0/256637472 >> 172.21.15.2:0/256637472 conn(0x7f21680f79e0 msgr2=0x7f21680f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.658+0000 7f216cc17700 1 -- 172.21.15.2:0/256637472 shutdown_connections 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.658+0000 7f216cc17700 1 -- 172.21.15.2:0/256637472 wait complete. 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216cc17700 1 Processor -- start 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216cc17700 1 -- start start 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216cc17700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2168101e50 0x7f21680fdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216cc17700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21681027f0 0x7f21680fe010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216cc17700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21680fe610 con 0x7f2168101e50 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216cc17700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21680fe780 con 0x7f21681027f0 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2168101e50 0x7f21680fdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2168101e50 0x7f21680fdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46786/0 (socket says 172.21.15.2:46786) 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216659c700 1 -- 172.21.15.2:0/1922018558 learned_addr learned my addr 172.21.15.2:0/1922018558 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:17:56.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f2165d9b700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21681027f0 0x7f21680fe010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:56.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216659c700 1 -- 172.21.15.2:0/1922018558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21681027f0 msgr2=0x7f21680fe010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:56.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216659c700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21681027f0 0x7f21680fe010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:56.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.659+0000 7f216659c700 1 -- 172.21.15.2:0/1922018558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150005040 con 0x7f2168101e50 2024-02-16T22:17:56.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.660+0000 7f216659c700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2168101e50 0x7f21680fdad0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2158002a50 tx=0x7f21580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:56.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.660+0000 7f215effd700 1 -- 172.21.15.2:0/1922018558 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2158007cc0 con 0x7f2168101e50 2024-02-16T22:17:56.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.660+0000 7f215effd700 1 -- 172.21.15.2:0/1922018558 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f215800f040 con 0x7f2168101e50 2024-02-16T22:17:56.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.660+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2168119cd0 con 0x7f2168101e50 2024-02-16T22:17:56.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.660+0000 7f215effd700 1 -- 172.21.15.2:0/1922018558 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f215800da30 con 0x7f2168101e50 2024-02-16T22:17:56.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.660+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216811a180 con 0x7f2168101e50 2024-02-16T22:17:56.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.661+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21680fb5e0 con 0x7f2168101e50 2024-02-16T22:17:56.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.662+0000 7f215effd700 1 -- 172.21.15.2:0/1922018558 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f215800db90 con 0x7f2168101e50 2024-02-16T22:17:56.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.662+0000 7f215effd700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f21540692a0 0x7f215406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:17:56.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.662+0000 7f215effd700 1 -- 172.21.15.2:0/1922018558 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2158086c00 con 0x7f2168101e50 2024-02-16T22:17:56.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.662+0000 7f2165d9b700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f21540692a0 0x7f215406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:17:56.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.663+0000 7f2165d9b700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f21540692a0 0x7f215406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f215000b3b0 tx=0x7f2150009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:17:56.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.671+0000 7f215effd700 1 -- 172.21.15.2:0/1922018558 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f215808c020 con 0x7f2168101e50 2024-02-16T22:17:56.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.865+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2168108f30 con 0x7f2168101e50 2024-02-16T22:17:56.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.866+0000 7f215effd700 1 -- 172.21.15.2:0/1922018558 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11003 (secure 0 0 0) 0x7f2158056c80 con 0x7f2168101e50 2024-02-16T22:17:56.868 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:17:56.868 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":40,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","created":"2024-02-16T22:11:59.286486+0000","modified":"2024-02-16T22:17:33.876912+0000","last_up_change":"2024-02-16T22:17:32.871791+0000","last_in_change":"2024-02-16T22:17:13.716253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-16T22:16:18.867363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"348575cf-dd0d-444c-bab9-bd51dcc9189c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6803","nonce":3509867548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6805","nonce":3509867548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6809","nonce":3509867548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6807","nonce":3509867548}]},"public_addr":"172.21.15.2:6803/3509867548","cluster_addr":"172.21.15.2:6805/3509867548","heartbeat_back_addr":"172.21.15.2:6809/3509867548","heartbeat_front_addr":"172.21.15.2:6807/3509867548","state":["exists","up"]},{"osd":1,"uuid":"eaffda16-c0db-440f-9a1a-99e228b3dc19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6811","nonce":609822393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6813","nonce":609822393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6817","nonce":609822393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6815","nonce":609822393}]},"public_addr":"172.21.15.2:6811/609822393","cluster_addr":"172.21.15.2:6813/609822393","heartbeat_back_addr":"172.21.15.2:6817/609822393","heartbeat_front_addr":"172.21.15.2:6815/609822393","state":["exists","up"]},{"osd":2,"uuid":"732c0a64-5183-42aa-8220-a12116e52442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6819","nonce":2737615042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6821","nonce":2737615042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6825","nonce":2737615042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6823","nonce":2737615042}]},"public_addr":"172.21.15.2:6819/2737615042","cluster_addr":"172.21.15.2:6821/2737615042","heartbeat_back_addr":"172.21.15.2:6825/2737615042","heartbeat_front_addr":"172.21.15.2:6823/2737615042","state":["exists","up"]},{"osd":3,"uuid":"09a182db-e441-45f8-9d13-61d69b7db54d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6801","nonce":425086385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6803","nonce":425086385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6807","nonce":425086385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6805","nonce":425086385}]},"public_addr":"172.21.15.103:6801/425086385","cluster_addr":"172.21.15.103:6803/425086385","heartbeat_back_addr":"172.21.15.103:6807/425086385","heartbeat_front_addr":"172.21.15.103:6805/425086385","state":["exists","up"]},{"osd":4,"uuid":"ee9a7426-801d-420f-be92-cd18c21d11b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6809","nonce":420978374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6811","nonce":420978374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6815","nonce":420978374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6813","nonce":420978374}]},"public_addr":"172.21.15.103:6809/420978374","cluster_addr":"172.21.15.103:6811/420978374","heartbeat_back_addr":"172.21.15.103:6815/420978374","heartbeat_front_addr":"172.21.15.103:6813/420978374","state":["exists","up"]},{"osd":5,"uuid":"68c72748-2d27-4ada-a35b-53e1bab20f9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6817","nonce":3241271941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6819","nonce":3241271941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6823","nonce":3241271941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6821","nonce":3241271941}]},"public_addr":"172.21.15.103:6817/3241271941","cluster_addr":"172.21.15.103:6819/3241271941","heartbeat_back_addr":"172.21.15.103:6823/3241271941","heartbeat_front_addr":"172.21.15.103:6821/3241271941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:15:21.168791+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:15:48.309630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:16:16.840477+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:16:40.949660+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:17:05.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:17:30.935853+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/1058268540":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6801/2481260931":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6800/2481260931":"2024-02-17T22:14:08.529242+0000","172.21.15.2:0/504770625":"2024-02-17T22:14:08.529242+0000","172.21.15.2:0/1364558863":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6801/673699505":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/135684194":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/3921669523":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/1318894983":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/3292480868":"2024-02-17T22:13:00.117629+0000","172.21.15.2:6801/395571706":"2024-02-17T22:12:27.545802+0000","172.21.15.2:6800/673699505":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/3834071444":"2024-02-17T22:12:27.545802+0000","172.21.15.2:6800/395571706":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/495154782":"2024-02-17T22:13:00.117629+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f21540692a0 msgr2=0x7f215406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f21540692a0 0x7f215406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f215000b3b0 tx=0x7f2150009e30 comp rx=0 tx=0).stop 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2168101e50 msgr2=0x7f21680fdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2168101e50 0x7f21680fdad0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2158002a50 tx=0x7f21580046b0 comp rx=0 tx=0).stop 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 shutdown_connections 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f21540692a0 0x7f215406b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2168101e50 0x7f21680fdad0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 --2- 172.21.15.2:0/1922018558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f21681027f0 0x7f21680fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 >> 172.21.15.2:0/1922018558 conn(0x7f21680f79e0 msgr2=0x7f2168066030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 shutdown_connections 2024-02-16T22:17:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:17:56.868+0000 7f216cc17700 1 -- 172.21.15.2:0/1922018558 wait complete. 2024-02-16T22:17:57.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:56 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1922018558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-16T22:17:57.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:56 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1922018558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-16T22:17:57.473 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-16T22:17:57.474 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-16T22:17:57.474 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-16T22:17:57.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-16T22:17:57.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-16T22:17:57.476 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-16T22:17:57.890 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:57.890 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:57.892 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:57 smithi002 ceph-mon[29154]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:58.145 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:58.145 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:58.145 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:58.146 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:17:58.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:57 smithi103 ceph-mon[36259]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:59.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:17:58 smithi002 ceph-mon[29154]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:17:59.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:17:58 smithi103 ceph-mon[36259]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:01 smithi002 ceph-mon[29154]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:02.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:01 smithi103 ceph-mon[36259]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:04.139 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:03 smithi002 ceph-mon[29154]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:04.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.246+0000 7f505d62c700 1 -- 172.21.15.2:0/2133846507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 msgr2=0x7f5058103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.246+0000 7f505d62c700 1 --2- 172.21.15.2:0/2133846507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 0x7f5058103320 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5048009ce0 tx=0x7f5048004d10 comp rx=0 tx=0).stop 2024-02-16T22:18:04.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.246+0000 7f505d62c700 1 -- 172.21.15.2:0/2133846507 shutdown_connections 2024-02-16T22:18:04.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.246+0000 7f505d62c700 1 --2- 172.21.15.2:0/2133846507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50581038f0 0x7f50580fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.246+0000 7f505d62c700 1 --2- 172.21.15.2:0/2133846507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 0x7f5058103320 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.246+0000 7f505d62c700 1 -- 172.21.15.2:0/2133846507 >> 172.21.15.2:0/2133846507 conn(0x7f50580f8ae0 msgr2=0x7f50580faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.246+0000 7f505d62c700 1 -- 172.21.15.2:0/2133846507 shutdown_connections 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.246+0000 7f505d62c700 1 -- 172.21.15.2:0/2133846507 wait complete. 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f505d62c700 1 Processor -- start 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f505d62c700 1 -- start start 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f505d62c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 0x7f5058113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f505d62c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50581038f0 0x7f5058114120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f505d62c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5058114770 con 0x7f5058102f50 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f505d62c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505810daa0 con 0x7f50581038f0 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f5056ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 0x7f5058113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f5056ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 0x7f5058113be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50204/0 (socket says 172.21.15.2:50204) 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f5056ffd700 1 -- 172.21.15.2:0/4086337748 learned_addr learned my addr 172.21.15.2:0/4086337748 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f50567fc700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50581038f0 0x7f5058114120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:04.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f5056ffd700 1 -- 172.21.15.2:0/4086337748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50581038f0 msgr2=0x7f5058114120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f5056ffd700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50581038f0 0x7f5058114120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f5056ffd700 1 -- 172.21.15.2:0/4086337748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048009940 con 0x7f5058102f50 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f5056ffd700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 0x7f5058113be0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f504800bf60 tx=0x7f5048004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.247+0000 7f504f7fe700 1 -- 172.21.15.2:0/4086337748 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504800e040 con 0x7f5058102f50 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.248+0000 7f504f7fe700 1 -- 172.21.15.2:0/4086337748 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50480128e0 con 0x7f5058102f50 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.248+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505810dd20 con 0x7f5058102f50 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.248+0000 7f504f7fe700 1 -- 172.21.15.2:0/4086337748 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504801c870 con 0x7f5058102f50 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.248+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505810e260 con 0x7f5058102f50 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.248+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f505810e740 con 0x7f5058102f50 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.249+0000 7f504f7fe700 1 -- 172.21.15.2:0/4086337748 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f5048012a50 con 0x7f5058102f50 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.249+0000 7f504f7fe700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f50440692a0 0x7f504406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.249+0000 7f504f7fe700 1 -- 172.21.15.2:0/4086337748 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5048088bc0 con 0x7f5058102f50 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.249+0000 7f50567fc700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f50440692a0 0x7f504406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.249+0000 7f504f7fe700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] conn(0x7f504406ef70 0x7f5044071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.249+0000 7f504f7fe700 1 -- 172.21.15.2:0/4086337748 --> [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5044071a30 con 0x7f504406ef70 2024-02-16T22:18:04.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.249+0000 7f504f7fe700 1 -- 172.21.15.2:0/4086337748 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f504808d220 con 0x7f5058102f50 2024-02-16T22:18:04.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.250+0000 7f50577fe700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] conn(0x7f504406ef70 0x7f5044071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:04.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.250+0000 7f50567fc700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f50440692a0 0x7f504406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5040007fd0 tx=0x7f5040009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:04.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.250+0000 7f50577fe700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] conn(0x7f504406ef70 0x7f5044071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:04.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.251+0000 7f504f7fe700 1 -- 172.21.15.2:0/4086337748 <== osd.0 v2:172.21.15.2:6802/3509867548 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5044071a30 con 0x7f504406ef70 2024-02-16T22:18:04.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.265+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 --> [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f505810e900 con 0x7f504406ef70 2024-02-16T22:18:04.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.265+0000 7f504f7fe700 1 -- 172.21.15.2:0/4086337748 <== osd.0 v2:172.21.15.2:6802/3509867548 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f505810e900 con 0x7f504406ef70 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] conn(0x7f504406ef70 msgr2=0x7f5044071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] conn(0x7f504406ef70 0x7f5044071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f50440692a0 msgr2=0x7f504406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f50440692a0 0x7f504406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5040007fd0 tx=0x7f5040009040 comp rx=0 tx=0).stop 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 msgr2=0x7f5058113be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 0x7f5058113be0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f504800bf60 tx=0x7f5048004d10 comp rx=0 tx=0).stop 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 shutdown_connections 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6802/3509867548,v1:172.21.15.2:6803/3509867548] conn(0x7f504406ef70 0x7f5044071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f50440692a0 0x7f504406b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5058102f50 0x7f5058113be0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 --2- 172.21.15.2:0/4086337748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50581038f0 0x7f5058114120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 >> 172.21.15.2:0/4086337748 conn(0x7f50580f8ae0 msgr2=0x7f50580fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 shutdown_connections 2024-02-16T22:18:04.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.266+0000 7f505d62c700 1 -- 172.21.15.2:0/4086337748 wait complete. 2024-02-16T22:18:04.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:03 smithi103 ceph-mon[36259]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:04.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.136+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/1449262384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98101e60 msgr2=0x7f1e98102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.136+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/1449262384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98101e60 0x7f1e98102230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1e88004940 tx=0x7f1e88009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:04.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.136+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/1449262384 shutdown_connections 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.136+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/1449262384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98102800 0x7f1e980fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.136+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/1449262384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98101e60 0x7f1e98102230 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.136+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/1449262384 >> 172.21.15.2:0/1449262384 conn(0x7f1e980f7a10 msgr2=0x7f1e980f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.136+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/1449262384 shutdown_connections 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.136+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/1449262384 wait complete. 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.137+0000 7f1e9dcf1700 1 Processor -- start 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.137+0000 7f1e9dcf1700 1 -- start start 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.137+0000 7f1e9dcf1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98101e60 0x7f1e98115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.137+0000 7f1e9dcf1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98102800 0x7f1e98116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.137+0000 7f1e9dcf1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e98116850 con 0x7f1e98102800 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.137+0000 7f1e9dcf1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9810fba0 con 0x7f1e98101e60 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e96ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98102800 0x7f1e98116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e96ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98102800 0x7f1e98116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50184/0 (socket says 172.21.15.2:50184) 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e96ffd700 1 -- 172.21.15.2:0/2734567314 learned_addr learned my addr 172.21.15.2:0/2734567314 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e977fe700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98101e60 0x7f1e98115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e96ffd700 1 -- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98101e60 msgr2=0x7f1e98115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e96ffd700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98101e60 0x7f1e98115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e96ffd700 1 -- 172.21.15.2:0/2734567314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e88005040 con 0x7f1e98102800 2024-02-16T22:18:04.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e96ffd700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98102800 0x7f1e98116200 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c002a50 tx=0x7f1e8c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:04.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e7ffff700 1 -- 172.21.15.2:0/2734567314 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e8c007cc0 con 0x7f1e98102800 2024-02-16T22:18:04.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e7ffff700 1 -- 172.21.15.2:0/2734567314 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e8c00f040 con 0x7f1e98102800 2024-02-16T22:18:04.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9810fe80 con 0x7f1e98102800 2024-02-16T22:18:04.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e7ffff700 1 -- 172.21.15.2:0/2734567314 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e8c00d9c0 con 0x7f1e98102800 2024-02-16T22:18:04.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.138+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e981103c0 con 0x7f1e98102800 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.139+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e981108e0 con 0x7f1e98102800 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.140+0000 7f1e7ffff700 1 -- 172.21.15.2:0/2734567314 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f1e8c009730 con 0x7f1e98102800 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.140+0000 7f1e7ffff700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1e800692a0 0x7f1e8006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.140+0000 7f1e7ffff700 1 -- 172.21.15.2:0/2734567314 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1e8c086bc0 con 0x7f1e98102800 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.140+0000 7f1e977fe700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1e800692a0 0x7f1e8006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.140+0000 7f1e7ffff700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] conn(0x7f1e8006ef70 0x7f1e80071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.140+0000 7f1e7ffff700 1 -- 172.21.15.2:0/2734567314 --> [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e80071a30 con 0x7f1e8006ef70 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.140+0000 7f1e7ffff700 1 -- 172.21.15.2:0/2734567314 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e8c08b220 con 0x7f1e98102800 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.140+0000 7f1e977fe700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1e800692a0 0x7f1e8006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e88004f90 tx=0x7f1e88009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.140+0000 7f1e97fff700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] conn(0x7f1e8006ef70 0x7f1e80071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.141+0000 7f1e97fff700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] conn(0x7f1e8006ef70 0x7f1e80071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.142+0000 7f1e7ffff700 1 -- 172.21.15.2:0/2734567314 <== osd.4 v2:172.21.15.103:6808/420978374 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f1e80071a30 con 0x7f1e8006ef70 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.158+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 --> [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1e98066a20 con 0x7f1e8006ef70 2024-02-16T22:18:04.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.158+0000 7f1e7ffff700 1 -- 172.21.15.2:0/2734567314 <== osd.4 v2:172.21.15.103:6808/420978374 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1e98066a20 con 0x7f1e8006ef70 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] conn(0x7f1e8006ef70 msgr2=0x7f1e80071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] conn(0x7f1e8006ef70 0x7f1e80071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1e800692a0 msgr2=0x7f1e8006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1e800692a0 0x7f1e8006b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1e88004f90 tx=0x7f1e88009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98102800 msgr2=0x7f1e98116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98102800 0x7f1e98116200 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c002a50 tx=0x7f1e8c004600 comp rx=0 tx=0).stop 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 shutdown_connections 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1e800692a0 0x7f1e8006b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e98101e60 0x7f1e98115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.103:6808/420978374,v1:172.21.15.103:6809/420978374] conn(0x7f1e8006ef70 0x7f1e80071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 --2- 172.21.15.2:0/2734567314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e98102800 0x7f1e98116200 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 >> 172.21.15.2:0/2734567314 conn(0x7f1e980f7a10 msgr2=0x7f1e98066050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 shutdown_connections 2024-02-16T22:18:04.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:04.159+0000 7f1e9dcf1700 1 -- 172.21.15.2:0/2734567314 wait complete. 2024-02-16T22:18:05.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.276+0000 7f57262c4700 1 -- 172.21.15.2:0/1125408379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fe8f0 msgr2=0x7f5720107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.276+0000 7f57262c4700 1 --2- 172.21.15.2:0/1125408379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fe8f0 0x7f5720107050 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5708004970 tx=0x7f5708009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:05.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.276+0000 7f57262c4700 1 -- 172.21.15.2:0/1125408379 shutdown_connections 2024-02-16T22:18:05.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.276+0000 7f57262c4700 1 --2- 172.21.15.2:0/1125408379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fe8f0 0x7f5720107050 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.276+0000 7f57262c4700 1 --2- 172.21.15.2:0/1125408379 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57200fdfe0 0x7f57200fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.276+0000 7f57262c4700 1 -- 172.21.15.2:0/1125408379 >> 172.21.15.2:0/1125408379 conn(0x7f5720071d90 msgr2=0x7f5720072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.276+0000 7f57262c4700 1 -- 172.21.15.2:0/1125408379 shutdown_connections 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.276+0000 7f57262c4700 1 -- 172.21.15.2:0/1125408379 wait complete. 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.276+0000 7f57262c4700 1 Processor -- start 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f57262c4700 1 -- start start 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f57262c4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fdfe0 0x7f57200fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f57262c4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57200fe8f0 0x7f57200fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f57262c4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57200fca70 con 0x7f57200fdfe0 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f57262c4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57200fcbe0 con 0x7f57200fe8f0 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f571ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fdfe0 0x7f57200fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f571ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fdfe0 0x7f57200fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50228/0 (socket says 172.21.15.2:50228) 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f571ffff700 1 -- 172.21.15.2:0/1248968329 learned_addr learned my addr 172.21.15.2:0/1248968329 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f571ffff700 1 -- 172.21.15.2:0/1248968329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57200fe8f0 msgr2=0x7f57200fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f571ffff700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57200fe8f0 0x7f57200fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f571ffff700 1 -- 172.21.15.2:0/1248968329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5708005040 con 0x7f57200fdfe0 2024-02-16T22:18:05.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f571ffff700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fdfe0 0x7f57200fbea0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f571000b790 tx=0x7f5710009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:05.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f571d7fa700 1 -- 172.21.15.2:0/1248968329 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f571000f770 con 0x7f57200fdfe0 2024-02-16T22:18:05.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f571d7fa700 1 -- 172.21.15.2:0/1248968329 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f571000fda0 con 0x7f57200fdfe0 2024-02-16T22:18:05.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.277+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5720119d10 con 0x7f57200fdfe0 2024-02-16T22:18:05.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.278+0000 7f571d7fa700 1 -- 172.21.15.2:0/1248968329 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f571000d310 con 0x7f57200fdfe0 2024-02-16T22:18:05.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.278+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572011a200 con 0x7f57200fdfe0 2024-02-16T22:18:05.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.278+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f572005e650 con 0x7f57200fdfe0 2024-02-16T22:18:05.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.279+0000 7f571d7fa700 1 -- 172.21.15.2:0/1248968329 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f5710010420 con 0x7f57200fdfe0 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.279+0000 7f571d7fa700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f570c0692b0 0x7f570c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.279+0000 7f571d7fa700 1 -- 172.21.15.2:0/1248968329 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5710087800 con 0x7f57200fdfe0 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.279+0000 7f5717fff700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f570c0692b0 0x7f570c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.280+0000 7f571d7fa700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] conn(0x7f570c06ef80 0x7f570c071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.280+0000 7f571d7fa700 1 -- 172.21.15.2:0/1248968329 --> [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f570c071a40 con 0x7f570c06ef80 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.280+0000 7f571d7fa700 1 -- 172.21.15.2:0/1248968329 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5710087c00 con 0x7f57200fdfe0 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.280+0000 7f5724861700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] conn(0x7f570c06ef80 0x7f570c071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.280+0000 7f5717fff700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f570c0692b0 0x7f570c06b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f570800b370 tx=0x7f5708009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.280+0000 7f5724861700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] conn(0x7f570c06ef80 0x7f570c071390 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:05.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.281+0000 7f571d7fa700 1 -- 172.21.15.2:0/1248968329 <== osd.1 v2:172.21.15.2:6810/609822393 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f570c071a40 con 0x7f570c06ef80 2024-02-16T22:18:05.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 --> [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f572005ea50 con 0x7f570c06ef80 2024-02-16T22:18:05.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f571d7fa700 1 -- 172.21.15.2:0/1248968329 <== osd.1 v2:172.21.15.2:6810/609822393 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f572005ea50 con 0x7f570c06ef80 2024-02-16T22:18:05.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] conn(0x7f570c06ef80 msgr2=0x7f570c071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] conn(0x7f570c06ef80 0x7f570c071390 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f570c0692b0 msgr2=0x7f570c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f570c0692b0 0x7f570c06b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f570800b370 tx=0x7f5708009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fdfe0 msgr2=0x7f57200fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fdfe0 0x7f57200fbea0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f571000b790 tx=0x7f5710009250 comp rx=0 tx=0).stop 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 shutdown_connections 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6810/609822393,v1:172.21.15.2:6811/609822393] conn(0x7f570c06ef80 0x7f570c071390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f570c0692b0 0x7f570c06b760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57200fdfe0 0x7f57200fbea0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 --2- 172.21.15.2:0/1248968329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57200fe8f0 0x7f57200fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 >> 172.21.15.2:0/1248968329 conn(0x7f5720071d90 msgr2=0x7f5720101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:05.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 shutdown_connections 2024-02-16T22:18:05.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.296+0000 7f57262c4700 1 -- 172.21.15.2:0/1248968329 wait complete. 2024-02-16T22:18:05.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.504+0000 7f29d2725700 1 -- 172.21.15.2:0/1987641696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 msgr2=0x7f29cc102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.504+0000 7f29d2725700 1 --2- 172.21.15.2:0/1987641696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 0x7f29cc102210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f29bc004940 tx=0x7f29bc009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:05.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.504+0000 7f29d2725700 1 -- 172.21.15.2:0/1987641696 shutdown_connections 2024-02-16T22:18:05.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.504+0000 7f29d2725700 1 --2- 172.21.15.2:0/1987641696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29cc1027e0 0x7f29cc0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.504+0000 7f29d2725700 1 --2- 172.21.15.2:0/1987641696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 0x7f29cc102210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.504+0000 7f29d2725700 1 -- 172.21.15.2:0/1987641696 >> 172.21.15.2:0/1987641696 conn(0x7f29cc071d90 msgr2=0x7f29cc072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:05.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.504+0000 7f29d2725700 1 -- 172.21.15.2:0/1987641696 shutdown_connections 2024-02-16T22:18:05.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.504+0000 7f29d2725700 1 -- 172.21.15.2:0/1987641696 wait complete. 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.504+0000 7f29d2725700 1 Processor -- start 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29d2725700 1 -- start start 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29d2725700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 0x7f29cc112960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29d2725700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29cc1027e0 0x7f29cc112ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29d2725700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc1134f0 con 0x7f29cc101e40 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29d2725700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc10c820 con 0x7f29cc1027e0 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29cbfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 0x7f29cc112960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29cbfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 0x7f29cc112960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50254/0 (socket says 172.21.15.2:50254) 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29cbfff700 1 -- 172.21.15.2:0/3298293893 learned_addr learned my addr 172.21.15.2:0/3298293893 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29cb7fe700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29cc1027e0 0x7f29cc112ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29cbfff700 1 -- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29cc1027e0 msgr2=0x7f29cc112ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29cbfff700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29cc1027e0 0x7f29cc112ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29cbfff700 1 -- 172.21.15.2:0/3298293893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc005040 con 0x7f29cc101e40 2024-02-16T22:18:05.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29cbfff700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 0x7f29cc112960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00b3b0 tx=0x7f29bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29c8ff9700 1 -- 172.21.15.2:0/3298293893 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29bc00e040 con 0x7f29cc101e40 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29c8ff9700 1 -- 172.21.15.2:0/3298293893 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29bc00bbf0 con 0x7f29cc101e40 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29c8ff9700 1 -- 172.21.15.2:0/3298293893 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29bc01c630 con 0x7f29cc101e40 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29cc10ca40 con 0x7f29cc101e40 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.505+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29cc10cf50 con 0x7f29cc101e40 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.506+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f29cc10d480 con 0x7f29cc101e40 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.507+0000 7f29c8ff9700 1 -- 172.21.15.2:0/3298293893 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f29bc00bd60 con 0x7f29cc101e40 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.507+0000 7f29c8ff9700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29b40692a0 0x7f29b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.507+0000 7f29c8ff9700 1 -- 172.21.15.2:0/3298293893 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f29bc088170 con 0x7f29cc101e40 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.507+0000 7f29cb7fe700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29b40692a0 0x7f29b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.507+0000 7f29c8ff9700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] conn(0x7f29b406ef70 0x7f29b4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.507+0000 7f29c8ff9700 1 -- 172.21.15.2:0/3298293893 --> [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29b4071a30 con 0x7f29b406ef70 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.507+0000 7f29c8ff9700 1 -- 172.21.15.2:0/3298293893 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29bc08a7e0 con 0x7f29cc101e40 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.508+0000 7f29d0cc2700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] conn(0x7f29b406ef70 0x7f29b4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.508+0000 7f29cb7fe700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29b40692a0 0x7f29b406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009fd0 tx=0x7f29c0009f20 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.508+0000 7f29d0cc2700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] conn(0x7f29b406ef70 0x7f29b4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:05.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.509+0000 7f29c8ff9700 1 -- 172.21.15.2:0/3298293893 <== osd.3 v2:172.21.15.103:6800/425086385 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f29b4071a30 con 0x7f29b406ef70 2024-02-16T22:18:05.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.523+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 --> [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f29cc0738b0 con 0x7f29b406ef70 2024-02-16T22:18:05.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.524+0000 7f29c8ff9700 1 -- 172.21.15.2:0/3298293893 <== osd.3 v2:172.21.15.103:6800/425086385 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f29cc0738b0 con 0x7f29b406ef70 2024-02-16T22:18:05.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.524+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] conn(0x7f29b406ef70 msgr2=0x7f29b4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.524+0000 7f29d2725700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] conn(0x7f29b406ef70 0x7f29b4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.524+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29b40692a0 msgr2=0x7f29b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.524+0000 7f29d2725700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29b40692a0 0x7f29b406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009fd0 tx=0x7f29c0009f20 comp rx=0 tx=0).stop 2024-02-16T22:18:05.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.524+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 msgr2=0x7f29cc112960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:05.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.524+0000 7f29d2725700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 0x7f29cc112960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00b3b0 tx=0x7f29bc009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:05.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.524+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 shutdown_connections 2024-02-16T22:18:05.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.525+0000 7f29d2725700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29b40692a0 0x7f29b406b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.525+0000 7f29d2725700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc101e40 0x7f29cc112960 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.525+0000 7f29d2725700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:6800/425086385,v1:172.21.15.103:6801/425086385] conn(0x7f29b406ef70 0x7f29b4071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.525+0000 7f29d2725700 1 --2- 172.21.15.2:0/3298293893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29cc1027e0 0x7f29cc112ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:05.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.525+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 >> 172.21.15.2:0/3298293893 conn(0x7f29cc071d90 msgr2=0x7f29cc0fadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:05.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.525+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 shutdown_connections 2024-02-16T22:18:05.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:05.525+0000 7f29d2725700 1 -- 172.21.15.2:0/3298293893 wait complete. 2024-02-16T22:18:06.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:05 smithi002 ceph-mon[29154]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:06.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:05 smithi103 ceph-mon[36259]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:06.592 INFO:teuthology.orchestra.run.smithi002.stdout:60129542174 2024-02-16T22:18:06.592 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-16T22:18:06.980 INFO:teuthology.orchestra.run.smithi002.stdout:38654705700 2024-02-16T22:18:06.980 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-16T22:18:07.391 INFO:teuthology.orchestra.run.smithi002.stdout:111669149716 2024-02-16T22:18:07.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-16T22:18:07.785 INFO:teuthology.orchestra.run.smithi002.stdout:146028888078 2024-02-16T22:18:07.785 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-16T22:18:08.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:07 smithi002 ceph-mon[29154]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:08.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:07 smithi103 ceph-mon[36259]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:08.954 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:18:09.227 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:08 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.421+0000 7ff6fc19b700 1 -- 172.21.15.2:0/3891859824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 msgr2=0x7ff6f40fe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.421+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/3891859824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 0x7ff6f40fe320 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4004940 tx=0x7ff6e4009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.421+0000 7ff6fc19b700 1 -- 172.21.15.2:0/3891859824 shutdown_connections 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.421+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/3891859824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f40fe8f0 0x7ff6f4106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.421+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/3891859824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 0x7ff6f40fe320 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.421+0000 7ff6fc19b700 1 -- 172.21.15.2:0/3891859824 >> 172.21.15.2:0/3891859824 conn(0x7ff6f40f7a10 msgr2=0x7ff6f40f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.421+0000 7ff6fc19b700 1 -- 172.21.15.2:0/3891859824 shutdown_connections 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.421+0000 7ff6fc19b700 1 -- 172.21.15.2:0/3891859824 wait complete. 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.422+0000 7ff6fc19b700 1 Processor -- start 2024-02-16T22:18:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.422+0000 7ff6fc19b700 1 -- start start 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.422+0000 7ff6fc19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 0x7ff6f4113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.422+0000 7ff6fc19b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f40fe8f0 0x7ff6f4113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.422+0000 7ff6fc19b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f41145b0 con 0x7ff6f40fdf50 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.422+0000 7ff6fc19b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f410d950 con 0x7ff6f40fe8f0 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.422+0000 7ff6f9f37700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 0x7ff6f4113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.422+0000 7ff6f9f37700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 0x7ff6f4113a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50272/0 (socket says 172.21.15.2:50272) 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.422+0000 7ff6f9f37700 1 -- 172.21.15.2:0/1493300189 learned_addr learned my addr 172.21.15.2:0/1493300189 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6f9736700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f40fe8f0 0x7ff6f4113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6f9f37700 1 -- 172.21.15.2:0/1493300189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f40fe8f0 msgr2=0x7ff6f4113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6f9f37700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f40fe8f0 0x7ff6f4113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6f9f37700 1 -- 172.21.15.2:0/1493300189 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4005040 con 0x7ff6f40fdf50 2024-02-16T22:18:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6f9f37700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 0x7ff6f4113a20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4004890 tx=0x7ff6e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6ea7fc700 1 -- 172.21.15.2:0/1493300189 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6e400e040 con 0x7ff6f40fdf50 2024-02-16T22:18:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6ea7fc700 1 -- 172.21.15.2:0/1493300189 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6e4004290 con 0x7ff6f40fdf50 2024-02-16T22:18:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f410dbd0 con 0x7ff6f40fdf50 2024-02-16T22:18:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6ea7fc700 1 -- 172.21.15.2:0/1493300189 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6e401c3f0 con 0x7ff6f40fdf50 2024-02-16T22:18:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.423+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f410e110 con 0x7ff6f40fdf50 2024-02-16T22:18:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.424+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff6f410e600 con 0x7ff6f40fdf50 2024-02-16T22:18:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.424+0000 7ff6ea7fc700 1 -- 172.21.15.2:0/1493300189 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff6e400bbf0 con 0x7ff6f40fdf50 2024-02-16T22:18:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.425+0000 7ff6ea7fc700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6e00692a0 0x7ff6e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.425+0000 7ff6ea7fc700 1 -- 172.21.15.2:0/1493300189 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff6e40880b0 con 0x7ff6f40fdf50 2024-02-16T22:18:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.425+0000 7ff6f9736700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6e00692a0 0x7ff6e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.425+0000 7ff6ea7fc700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] conn(0x7ff6e006ef70 0x7ff6e0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.425+0000 7ff6ea7fc700 1 -- 172.21.15.2:0/1493300189 --> [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6e0071a30 con 0x7ff6e006ef70 2024-02-16T22:18:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.425+0000 7ff6ea7fc700 1 -- 172.21.15.2:0/1493300189 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6e408a720 con 0x7ff6f40fdf50 2024-02-16T22:18:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.425+0000 7ff6fa738700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] conn(0x7ff6e006ef70 0x7ff6e0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.425+0000 7ff6f9736700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6e00692a0 0x7ff6e006b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff6f410eb60 tx=0x7ff6f000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.426+0000 7ff6fa738700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] conn(0x7ff6e006ef70 0x7ff6e0071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.426+0000 7ff6ea7fc700 1 -- 172.21.15.2:0/1493300189 <== osd.2 v2:172.21.15.2:6818/2737615042 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff6e0071a30 con 0x7ff6e006ef70 2024-02-16T22:18:09.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.441+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 --> [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff6f405e690 con 0x7ff6e006ef70 2024-02-16T22:18:09.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.441+0000 7ff6ea7fc700 1 -- 172.21.15.2:0/1493300189 <== osd.2 v2:172.21.15.2:6818/2737615042 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff6f405e690 con 0x7ff6e006ef70 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.441+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] conn(0x7ff6e006ef70 msgr2=0x7ff6e0071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.441+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] conn(0x7ff6e006ef70 0x7ff6e0071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6e00692a0 msgr2=0x7ff6e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6e00692a0 0x7ff6e006b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff6f410eb60 tx=0x7ff6f000b040 comp rx=0 tx=0).stop 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 msgr2=0x7ff6f4113a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 0x7ff6f4113a20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4004890 tx=0x7ff6e4009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 shutdown_connections 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff6e00692a0 0x7ff6e006b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6f40fdf50 0x7ff6f4113a20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.2:6818/2737615042,v1:172.21.15.2:6819/2737615042] conn(0x7ff6e006ef70 0x7ff6e0071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 --2- 172.21.15.2:0/1493300189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6f40fe8f0 0x7ff6f4113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 >> 172.21.15.2:0/1493300189 conn(0x7ff6f40f7a10 msgr2=0x7ff6f41017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 shutdown_connections 2024-02-16T22:18:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.442+0000 7ff6fc19b700 1 -- 172.21.15.2:0/1493300189 wait complete. 2024-02-16T22:18:09.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.637+0000 7f26c3e43700 1 -- 172.21.15.2:0/4234193068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 msgr2=0x7f26bc107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.637+0000 7f26c3e43700 1 --2- 172.21.15.2:0/4234193068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 0x7f26bc107050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f26ac004970 tx=0x7f26ac009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:09.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.637+0000 7f26c3e43700 1 -- 172.21.15.2:0/4234193068 shutdown_connections 2024-02-16T22:18:09.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.637+0000 7f26c3e43700 1 --2- 172.21.15.2:0/4234193068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 0x7f26bc107050 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.637+0000 7f26c3e43700 1 --2- 172.21.15.2:0/4234193068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26bc0fdfe0 0x7f26bc0fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.637+0000 7f26c3e43700 1 -- 172.21.15.2:0/4234193068 >> 172.21.15.2:0/4234193068 conn(0x7f26bc071d90 msgr2=0x7f26bc072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:09.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.638+0000 7f26c3e43700 1 -- 172.21.15.2:0/4234193068 shutdown_connections 2024-02-16T22:18:09.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.638+0000 7f26c3e43700 1 -- 172.21.15.2:0/4234193068 wait complete. 2024-02-16T22:18:09.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.638+0000 7f26c3e43700 1 Processor -- start 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.638+0000 7f26c3e43700 1 -- start start 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.638+0000 7f26c3e43700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26bc0fdfe0 0x7f26bc113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.638+0000 7f26c3e43700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 0x7f26bc113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.638+0000 7f26c3e43700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc114690 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.638+0000 7f26c3e43700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26bc10d930 con 0x7f26bc0fdfe0 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.638+0000 7f26c13de700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 0x7f26bc113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26c13de700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 0x7f26bc113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50286/0 (socket says 172.21.15.2:50286) 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26c13de700 1 -- 172.21.15.2:0/528236481 learned_addr learned my addr 172.21.15.2:0/528236481 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26c1bdf700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26bc0fdfe0 0x7f26bc113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26c13de700 1 -- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26bc0fdfe0 msgr2=0x7f26bc113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26c13de700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26bc0fdfe0 0x7f26bc113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26c13de700 1 -- 172.21.15.2:0/528236481 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac005040 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26c13de700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 0x7f26bc113fb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f26ac000c00 tx=0x7f26ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:09.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26b27fc700 1 -- 172.21.15.2:0/528236481 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26ac00e040 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26bc10dbb0 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26b27fc700 1 -- 172.21.15.2:0/528236481 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26ac009af0 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26bc10e0f0 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.639+0000 7f26b27fc700 1 -- 172.21.15.2:0/528236481 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26ac01c3f0 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.640+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f26bc10e6a0 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.641+0000 7f26b27fc700 1 -- 172.21.15.2:0/528236481 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f26ac00bc10 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.641+0000 7f26b27fc700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f26a80692a0 0x7f26a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:09.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.641+0000 7f26b27fc700 1 -- 172.21.15.2:0/528236481 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f26ac088b30 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.641+0000 7f26c1bdf700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f26a80692a0 0x7f26a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:09.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.641+0000 7f26b27fc700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] conn(0x7f26a806ef70 0x7f26a8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:09.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.641+0000 7f26b27fc700 1 -- 172.21.15.2:0/528236481 --> [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26a8071a30 con 0x7f26a806ef70 2024-02-16T22:18:09.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.641+0000 7f26b27fc700 1 -- 172.21.15.2:0/528236481 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26ac088f10 con 0x7f26bc0fe8f0 2024-02-16T22:18:09.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.641+0000 7f26c23e0700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] conn(0x7f26a806ef70 0x7f26a8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:09.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.641+0000 7f26c1bdf700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f26a80692a0 0x7f26a806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009bf0 tx=0x7f26b80091a0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:09.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.642+0000 7f26c23e0700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] conn(0x7f26a806ef70 0x7f26a8071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:09.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.642+0000 7f26b27fc700 1 -- 172.21.15.2:0/528236481 <== osd.5 v2:172.21.15.103:6816/3241271941 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f26a8071a30 con 0x7f26a806ef70 2024-02-16T22:18:09.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.657+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 --> [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f26bc05e610 con 0x7f26a806ef70 2024-02-16T22:18:09.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26b27fc700 1 -- 172.21.15.2:0/528236481 <== osd.5 v2:172.21.15.103:6816/3241271941 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f26bc05e610 con 0x7f26a806ef70 2024-02-16T22:18:09.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] conn(0x7f26a806ef70 msgr2=0x7f26a8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] conn(0x7f26a806ef70 0x7f26a8071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f26a80692a0 msgr2=0x7f26a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f26a80692a0 0x7f26a806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009bf0 tx=0x7f26b80091a0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 msgr2=0x7f26bc113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:09.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 0x7f26bc113fb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f26ac000c00 tx=0x7f26ac009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:09.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 shutdown_connections 2024-02-16T22:18:09.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:6816/3241271941,v1:172.21.15.103:6817/3241271941] conn(0x7f26a806ef70 0x7f26a8071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f26a80692a0 0x7f26a806b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26bc0fdfe0 0x7f26bc113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 --2- 172.21.15.2:0/528236481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26bc0fe8f0 0x7f26bc113fb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:09.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 >> 172.21.15.2:0/528236481 conn(0x7f26bc071d90 msgr2=0x7f26bc101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:09.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 shutdown_connections 2024-02-16T22:18:09.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:09.658+0000 7f26c3e43700 1 -- 172.21.15.2:0/528236481 wait complete. 2024-02-16T22:18:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:09 smithi002 ceph-mon[29154]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:10.316 INFO:teuthology.orchestra.run.smithi002.stdout:81604378648 2024-02-16T22:18:10.316 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-16T22:18:10.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:09 smithi103 ceph-mon[36259]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:11.017 INFO:teuthology.orchestra.run.smithi002.stdout:167503724555 2024-02-16T22:18:11.017 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-16T22:18:11.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:10 smithi002 ceph-mon[29154]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:11.284 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:11.285 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:11.285 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:11.285 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:11.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:10 smithi103 ceph-mon[36259]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:11.339 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:13.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:13 smithi002 ceph-mon[29154]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:14.064 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:14.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:13 smithi103 ceph-mon[36259]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:16.187 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:15 smithi002 ceph-mon[29154]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:15 smithi103 ceph-mon[36259]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:18.142 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:17 smithi002 ceph-mon[29154]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:18.143 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:17 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3821536629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-16T22:18:18.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:17 smithi103 ceph-mon[36259]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:18.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:17 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3821536629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-16T22:18:18.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.734+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3203462834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 msgr2=0x7ff2780fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:18.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.734+0000 7ff27f5c7700 1 --2- 172.21.15.2:0/3203462834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 0x7ff2780fbea0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff26c004940 tx=0x7ff26c009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.734+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3203462834 shutdown_connections 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.734+0000 7ff27f5c7700 1 --2- 172.21.15.2:0/3203462834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 0x7ff2780fbea0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.734+0000 7ff27f5c7700 1 --2- 172.21.15.2:0/3203462834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff278101e40 0x7ff278102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.734+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3203462834 >> 172.21.15.2:0/3203462834 conn(0x7ff278071d90 msgr2=0x7ff278072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.734+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3203462834 shutdown_connections 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.734+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3203462834 wait complete. 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27f5c7700 1 Processor -- start 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27f5c7700 1 -- start start 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27f5c7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff278101e40 0x7ff278102ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27f5c7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 0x7ff2781031e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:18.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27f5c7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff278103870 con 0x7ff2781027e0 2024-02-16T22:18:18.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27f5c7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2781039e0 con 0x7ff278101e40 2024-02-16T22:18:18.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27cb62700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 0x7ff2781031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:18.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27cb62700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 0x7ff2781031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33568/0 (socket says 172.21.15.2:33568) 2024-02-16T22:18:18.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27cb62700 1 -- 172.21.15.2:0/3821536629 learned_addr learned my addr 172.21.15.2:0/3821536629 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:18.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27d363700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff278101e40 0x7ff278102ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:18.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27cb62700 1 -- 172.21.15.2:0/3821536629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff278101e40 msgr2=0x7ff278102ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:18.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27cb62700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff278101e40 0x7ff278102ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:18.573 INFO:teuthology.orchestra.run.smithi002.stdout:60129542176 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27cb62700 1 -- 172.21.15.2:0/3821536629 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c005040 con 0x7ff2781027e0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.735+0000 7ff27cb62700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 0x7ff2781031e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0090d0 tx=0x7ff26c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.736+0000 7ff275ffb700 1 -- 172.21.15.2:0/3821536629 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff26c00e040 con 0x7ff2781027e0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.736+0000 7ff275ffb700 1 -- 172.21.15.2:0/3821536629 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff26c00bbf0 con 0x7ff2781027e0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.736+0000 7ff275ffb700 1 -- 172.21.15.2:0/3821536629 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff26c01c630 con 0x7ff2781027e0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.736+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff278119cc0 con 0x7ff2781027e0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.736+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27811a1b0 con 0x7ff2781027e0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.737+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2780738b0 con 0x7ff2781027e0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.737+0000 7ff275ffb700 1 -- 172.21.15.2:0/3821536629 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff26c00bd60 con 0x7ff2781027e0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.738+0000 7ff275ffb700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff2600692f0 0x7ff26006b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.738+0000 7ff27d363700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff2600692f0 0x7ff26006b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.738+0000 7ff275ffb700 1 -- 172.21.15.2:0/3821536629 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff26c088290 con 0x7ff2781027e0 2024-02-16T22:18:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.738+0000 7ff27d363700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff2600692f0 0x7ff26006b7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff268009f50 tx=0x7ff268009590 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.742+0000 7ff275ffb700 1 -- 172.21.15.2:0/3821536629 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff26c054860 con 0x7ff2781027e0 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.936+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff278108f90 con 0x7ff2781027e0 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.937+0000 7ff275ffb700 1 -- 172.21.15.2:0/3821536629 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff26c013e60 con 0x7ff2781027e0 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff2600692f0 msgr2=0x7ff26006b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff2600692f0 0x7ff26006b7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff268009f50 tx=0x7ff268009590 comp rx=0 tx=0).stop 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 msgr2=0x7ff2781031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 0x7ff2781031e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff26c0090d0 tx=0x7ff26c009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 shutdown_connections 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff2600692f0 0x7ff26006b7a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff278101e40 0x7ff278102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 --2- 172.21.15.2:0/3821536629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff2781027e0 0x7ff2781031e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 >> 172.21.15.2:0/3821536629 conn(0x7ff278071d90 msgr2=0x7ff278065740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 shutdown_connections 2024-02-16T22:18:18.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:16.939+0000 7ff27f5c7700 1 -- 172.21.15.2:0/3821536629 wait complete. 2024-02-16T22:18:19.100 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:18 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1809190908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-16T22:18:19.134 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-02-16T22:18:19.134 DEBUG:teuthology.parallel:result is None 2024-02-16T22:18:19.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:18 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1809190908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-16T22:18:19.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.568+0000 7f9ba7b61700 1 -- 172.21.15.2:0/462803194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba00fb570 msgr2=0x7f9ba00fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:19.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.568+0000 7f9ba7b61700 1 --2- 172.21.15.2:0/462803194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba00fb570 0x7f9ba00fb9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9b90004940 tx=0x7f9b90009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:19.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.569+0000 7f9ba7b61700 1 -- 172.21.15.2:0/462803194 shutdown_connections 2024-02-16T22:18:19.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.569+0000 7f9ba7b61700 1 --2- 172.21.15.2:0/462803194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba00fb570 0x7f9ba00fb9c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.569+0000 7f9ba7b61700 1 --2- 172.21.15.2:0/462803194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ba0103140 0x7f9ba0103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.569+0000 7f9ba7b61700 1 -- 172.21.15.2:0/462803194 >> 172.21.15.2:0/462803194 conn(0x7f9ba0071560 msgr2=0x7f9ba0071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:19.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.569+0000 7f9ba7b61700 1 -- 172.21.15.2:0/462803194 shutdown_connections 2024-02-16T22:18:19.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.569+0000 7f9ba7b61700 1 -- 172.21.15.2:0/462803194 wait complete. 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.569+0000 7f9ba7b61700 1 Processor -- start 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.569+0000 7f9ba7b61700 1 -- start start 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba7b61700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ba00fb570 0x7f9ba0113ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba7b61700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba0103140 0x7f9ba01140e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba7b61700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba01147c0 con 0x7f9ba0103140 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba7b61700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba010da60 con 0x7f9ba00fb570 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba50fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba0103140 0x7f9ba01140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba50fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba0103140 0x7f9ba01140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33610/0 (socket says 172.21.15.2:33610) 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba50fc700 1 -- 172.21.15.2:0/2011732536 learned_addr learned my addr 172.21.15.2:0/2011732536 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:19.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba58fd700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ba00fb570 0x7f9ba0113ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba50fc700 1 -- 172.21.15.2:0/2011732536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ba00fb570 msgr2=0x7f9ba0113ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba50fc700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ba00fb570 0x7f9ba0113ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba50fc700 1 -- 172.21.15.2:0/2011732536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b90005040 con 0x7f9ba0103140 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba50fc700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba0103140 0x7f9ba01140e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000b3b0 tx=0x7f9b90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9b967fc700 1 -- 172.21.15.2:0/2011732536 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b9000e040 con 0x7f9ba0103140 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9b967fc700 1 -- 172.21.15.2:0/2011732536 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b9000bbf0 con 0x7f9ba0103140 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba010dce0 con 0x7f9ba0103140 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.570+0000 7f9b967fc700 1 -- 172.21.15.2:0/2011732536 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b9001c3f0 con 0x7f9ba0103140 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.571+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba010e220 con 0x7f9ba0103140 2024-02-16T22:18:19.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.571+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba0105570 con 0x7f9ba0103140 2024-02-16T22:18:19.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.572+0000 7f9b967fc700 1 -- 172.21.15.2:0/2011732536 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f9b9000bd60 con 0x7f9ba0103140 2024-02-16T22:18:19.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.572+0000 7f9b967fc700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9b8c0692a0 0x7f9b8c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:19.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.572+0000 7f9b967fc700 1 -- 172.21.15.2:0/2011732536 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9b900880c0 con 0x7f9ba0103140 2024-02-16T22:18:19.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.572+0000 7f9ba58fd700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9b8c0692a0 0x7f9b8c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:19.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.573+0000 7f9ba58fd700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9b8c0692a0 0x7f9b8c06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009bf0 tx=0x7f9b9c0091a0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:19.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.577+0000 7f9b967fc700 1 -- 172.21.15.2:0/2011732536 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9b90058260 con 0x7f9ba0103140 2024-02-16T22:18:19.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.771+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9ba010f180 con 0x7f9ba0103140 2024-02-16T22:18:19.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.772+0000 7f9b967fc700 1 -- 172.21.15.2:0/2011732536 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9b90057df0 con 0x7f9ba0103140 2024-02-16T22:18:19.774 INFO:teuthology.orchestra.run.smithi002.stdout:146028888081 2024-02-16T22:18:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9b8c0692a0 msgr2=0x7f9b8c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9b8c0692a0 0x7f9b8c06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009bf0 tx=0x7f9b9c0091a0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba0103140 msgr2=0x7f9ba01140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba0103140 0x7f9ba01140e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000b3b0 tx=0x7f9b90009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 shutdown_connections 2024-02-16T22:18:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9b8c0692a0 0x7f9b8c06b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9ba00fb570 0x7f9ba0113ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 --2- 172.21.15.2:0/2011732536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ba0103140 0x7f9ba01140e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 >> 172.21.15.2:0/2011732536 conn(0x7f9ba0071560 msgr2=0x7f9ba00f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 shutdown_connections 2024-02-16T22:18:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.774+0000 7f9ba7b61700 1 -- 172.21.15.2:0/2011732536 wait complete. 2024-02-16T22:18:19.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.777+0000 7f32410fd700 1 -- 172.21.15.2:0/396724294 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c1027f0 msgr2=0x7f323c0fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:19.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.777+0000 7f32410fd700 1 --2- 172.21.15.2:0/396724294 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c1027f0 0x7f323c0fbe60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3230004940 tx=0x7f3230009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:19.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.777+0000 7f32410fd700 1 -- 172.21.15.2:0/396724294 shutdown_connections 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.777+0000 7f32410fd700 1 --2- 172.21.15.2:0/396724294 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c1027f0 0x7f323c0fbe60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.777+0000 7f32410fd700 1 --2- 172.21.15.2:0/396724294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f323c101e50 0x7f323c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.777+0000 7f32410fd700 1 -- 172.21.15.2:0/396724294 >> 172.21.15.2:0/396724294 conn(0x7f323c0f79e0 msgr2=0x7f323c0f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.777+0000 7f32410fd700 1 -- 172.21.15.2:0/396724294 shutdown_connections 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.777+0000 7f32410fd700 1 -- 172.21.15.2:0/396724294 wait complete. 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f32410fd700 1 Processor -- start 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f32410fd700 1 -- start start 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f32410fd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c101e50 0x7f323c112990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f32410fd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f323c1027f0 0x7f323c112ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f32410fd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c113520 con 0x7f323c101e50 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f32410fd700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c10c850 con 0x7f323c1027f0 2024-02-16T22:18:19.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f323ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c101e50 0x7f323c112990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f323ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c101e50 0x7f323c112990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33640/0 (socket says 172.21.15.2:33640) 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f323ad9d700 1 -- 172.21.15.2:0/359913761 learned_addr learned my addr 172.21.15.2:0/359913761 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f323a59c700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f323c1027f0 0x7f323c112ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f323ad9d700 1 -- 172.21.15.2:0/359913761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f323c1027f0 msgr2=0x7f323c112ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f323ad9d700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f323c1027f0 0x7f323c112ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.778+0000 7f323ad9d700 1 -- 172.21.15.2:0/359913761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3230005040 con 0x7f323c101e50 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.779+0000 7f323ad9d700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c101e50 0x7f323c112990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f322c002a50 tx=0x7f322c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.779+0000 7f32237fe700 1 -- 172.21.15.2:0/359913761 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c007cc0 con 0x7f323c101e50 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.779+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c10cb30 con 0x7f323c101e50 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.779+0000 7f32237fe700 1 -- 172.21.15.2:0/359913761 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f322c00f040 con 0x7f323c101e50 2024-02-16T22:18:19.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.779+0000 7f32237fe700 1 -- 172.21.15.2:0/359913761 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c009640 con 0x7f323c101e50 2024-02-16T22:18:19.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.779+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c10d070 con 0x7f323c101e50 2024-02-16T22:18:19.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.780+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c104320 con 0x7f323c101e50 2024-02-16T22:18:19.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.780+0000 7f32237fe700 1 -- 172.21.15.2:0/359913761 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f322c0097a0 con 0x7f323c101e50 2024-02-16T22:18:19.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.781+0000 7f32237fe700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32240692b0 0x7f322406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:19.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.781+0000 7f32237fe700 1 -- 172.21.15.2:0/359913761 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f322c086c70 con 0x7f323c101e50 2024-02-16T22:18:19.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.781+0000 7f323a59c700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32240692b0 0x7f322406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:19.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.781+0000 7f323a59c700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32240692b0 0x7f322406b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f323000b3b0 tx=0x7f3230009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:19.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.785+0000 7f32237fe700 1 -- 172.21.15.2:0/359913761 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f322c056cf0 con 0x7f323c101e50 2024-02-16T22:18:19.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:19 smithi002 ceph-mon[29154]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:19.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:19 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2011732536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-16T22:18:19.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.980+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f323c064b00 con 0x7f323c101e50 2024-02-16T22:18:19.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.981+0000 7f32237fe700 1 -- 172.21.15.2:0/359913761 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f322c056880 con 0x7f323c101e50 2024-02-16T22:18:19.982 INFO:teuthology.orchestra.run.smithi002.stdout:38654705702 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32240692b0 msgr2=0x7f322406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32240692b0 0x7f322406b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f323000b3b0 tx=0x7f3230009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c101e50 msgr2=0x7f323c112990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c101e50 0x7f323c112990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f322c002a50 tx=0x7f322c0046b0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 shutdown_connections 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f32240692b0 0x7f322406b760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f323c101e50 0x7f323c112990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 --2- 172.21.15.2:0/359913761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f323c1027f0 0x7f323c112ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 >> 172.21.15.2:0/359913761 conn(0x7f323c0f79e0 msgr2=0x7f323c0f94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:19.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 shutdown_connections 2024-02-16T22:18:19.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:19.983+0000 7f32410fd700 1 -- 172.21.15.2:0/359913761 wait complete. 2024-02-16T22:18:20.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:19 smithi103 ceph-mon[36259]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:20.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:19 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2011732536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-16T22:18:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.613+0000 7fe600fc6700 1 -- 172.21.15.2:0/248811707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 msgr2=0x7fe5fc0fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.613+0000 7fe600fc6700 1 --2- 172.21.15.2:0/248811707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 0x7fe5fc0fc0f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec004940 tx=0x7fe5ec009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.613+0000 7fe600fc6700 1 -- 172.21.15.2:0/248811707 shutdown_connections 2024-02-16T22:18:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.613+0000 7fe600fc6700 1 --2- 172.21.15.2:0/248811707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5fc0fc630 0x7fe5fc109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.613+0000 7fe600fc6700 1 --2- 172.21.15.2:0/248811707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 0x7fe5fc0fc0f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.613+0000 7fe600fc6700 1 -- 172.21.15.2:0/248811707 >> 172.21.15.2:0/248811707 conn(0x7fe5fc0f7a10 msgr2=0x7fe5fc0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:20.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.613+0000 7fe600fc6700 1 -- 172.21.15.2:0/248811707 shutdown_connections 2024-02-16T22:18:20.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe600fc6700 1 -- 172.21.15.2:0/248811707 wait complete. 2024-02-16T22:18:20.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe600fc6700 1 Processor -- start 2024-02-16T22:18:20.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe600fc6700 1 -- start start 2024-02-16T22:18:20.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe600fc6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 0x7fe5fc115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:20.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe600fc6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5fc0fc630 0x7fe5fc116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:20.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe600fc6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc116760 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe600fc6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc10fa90 con 0x7fe5fc0fc630 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe5fa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 0x7fe5fc115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe5fa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 0x7fe5fc115bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50684/0 (socket says 172.21.15.2:50684) 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.614+0000 7fe5fa59c700 1 -- 172.21.15.2:0/3221223917 learned_addr learned my addr 172.21.15.2:0/3221223917 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.615+0000 7fe5fa59c700 1 -- 172.21.15.2:0/3221223917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5fc0fc630 msgr2=0x7fe5fc116110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.615+0000 7fe5fa59c700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5fc0fc630 0x7fe5fc116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.615+0000 7fe5fa59c700 1 -- 172.21.15.2:0/3221223917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec005040 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.615+0000 7fe5fa59c700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 0x7fe5fc115bd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009180 tx=0x7fe5ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.615+0000 7fe5eaffd700 1 -- 172.21.15.2:0/3221223917 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5ec00e040 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.615+0000 7fe5eaffd700 1 -- 172.21.15.2:0/3221223917 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5ec00bbf0 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.615+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc10fd70 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.615+0000 7fe5eaffd700 1 -- 172.21.15.2:0/3221223917 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5ec01c630 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.615+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc1102b0 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.616+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc106ac0 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.616+0000 7fe5eaffd700 1 -- 172.21.15.2:0/3221223917 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe5ec00bd60 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.617+0000 7fe5eaffd700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5e40692b0 0x7fe5e406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.617+0000 7fe5eaffd700 1 -- 172.21.15.2:0/3221223917 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe5ec0881b0 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.617+0000 7fe5f9d9b700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5e40692b0 0x7fe5e406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.617+0000 7fe5f9d9b700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5e40692b0 0x7fe5e406b760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc111400 tx=0x7fe5f0007d40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:20.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.621+0000 7fe5eaffd700 1 -- 172.21.15.2:0/3221223917 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe5ec058350 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.811+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe5fc04bed0 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.812+0000 7fe5eaffd700 1 -- 172.21.15.2:0/3221223917 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe5ec057ee0 con 0x7fe5fc0fbd20 2024-02-16T22:18:20.814 INFO:teuthology.orchestra.run.smithi002.stdout:111669149718 2024-02-16T22:18:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5e40692b0 msgr2=0x7fe5e406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5e40692b0 0x7fe5e406b760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc111400 tx=0x7fe5f0007d40 comp rx=0 tx=0).stop 2024-02-16T22:18:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 msgr2=0x7fe5fc115bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 0x7fe5fc115bd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009180 tx=0x7fe5ec009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 shutdown_connections 2024-02-16T22:18:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5e40692b0 0x7fe5e406b760 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:20.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5fc0fbd20 0x7fe5fc115bd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:20.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 --2- 172.21.15.2:0/3221223917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5fc0fc630 0x7fe5fc116110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:20.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 >> 172.21.15.2:0/3221223917 conn(0x7fe5fc0f7a10 msgr2=0x7fe5fc0f90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:20.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 shutdown_connections 2024-02-16T22:18:20.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:20.814+0000 7fe600fc6700 1 -- 172.21.15.2:0/3221223917 wait complete. 2024-02-16T22:18:21.081 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:20 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/359913761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-16T22:18:21.082 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:20 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3221223917' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-16T22:18:21.104 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888078 got 146028888081 for osd.4 2024-02-16T22:18:21.104 DEBUG:teuthology.parallel:result is None 2024-02-16T22:18:21.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:20 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/359913761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-16T22:18:21.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:20 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3221223917' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-16T22:18:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.140+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1220951289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0101de0 msgr2=0x7f8ea01021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.140+0000 7f8ea6c90700 1 --2- 172.21.15.2:0/1220951289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0101de0 0x7f8ea01021b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8e94004940 tx=0x7f8e94009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.140+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1220951289 shutdown_connections 2024-02-16T22:18:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.140+0000 7f8ea6c90700 1 --2- 172.21.15.2:0/1220951289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea0102780 0x7f8ea00fbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.140+0000 7f8ea6c90700 1 --2- 172.21.15.2:0/1220951289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0101de0 0x7f8ea01021b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.140+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1220951289 >> 172.21.15.2:0/1220951289 conn(0x7f8ea0071d50 msgr2=0x7f8ea0072150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.140+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1220951289 shutdown_connections 2024-02-16T22:18:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.140+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1220951289 wait complete. 2024-02-16T22:18:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8ea6c90700 1 Processor -- start 2024-02-16T22:18:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8ea6c90700 1 -- start start 2024-02-16T22:18:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8ea6c90700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea0101de0 0x7f8ea0102f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8ea6c90700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0102780 0x7f8ea0103490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8ea6c90700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea0103b20 con 0x7f8ea0102780 2024-02-16T22:18:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8ea6c90700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea0103c90 con 0x7f8ea0101de0 2024-02-16T22:18:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8e9ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0102780 0x7f8ea0103490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8e9ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0102780 0x7f8ea0103490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33592/0 (socket says 172.21.15.2:33592) 2024-02-16T22:18:21.803 INFO:teuthology.orchestra.run.smithi002.stdout:167503724556 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8e9ffff700 1 -- 172.21.15.2:0/1809190908 learned_addr learned my addr 172.21.15.2:0/1809190908 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.141+0000 7f8ea4a2c700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea0101de0 0x7f8ea0102f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.142+0000 7f8e9ffff700 1 -- 172.21.15.2:0/1809190908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea0101de0 msgr2=0x7f8ea0102f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.142+0000 7f8e9ffff700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea0101de0 0x7f8ea0102f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.142+0000 7f8e9ffff700 1 -- 172.21.15.2:0/1809190908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e94005040 con 0x7f8ea0102780 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.142+0000 7f8e9ffff700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0102780 0x7f8ea0103490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00b790 tx=0x7f8e8c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.142+0000 7f8e9d7fa700 1 -- 172.21.15.2:0/1809190908 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e8c00f6f0 con 0x7f8ea0102780 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.142+0000 7f8e9d7fa700 1 -- 172.21.15.2:0/1809190908 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e8c00fd20 con 0x7f8ea0102780 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.142+0000 7f8e9d7fa700 1 -- 172.21.15.2:0/1809190908 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e8c0103a0 con 0x7f8ea0102780 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.142+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea0119c50 con 0x7f8ea0102780 2024-02-16T22:18:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.142+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea011a140 con 0x7f8ea0102780 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.143+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea0075ca0 con 0x7f8ea0102780 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.143+0000 7f8e9d7fa700 1 -- 172.21.15.2:0/1809190908 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f8e8c011400 con 0x7f8ea0102780 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.144+0000 7f8e9d7fa700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f8e900692b0 0x7f8e9006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.144+0000 7f8e9d7fa700 1 -- 172.21.15.2:0/1809190908 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8e8c086bf0 con 0x7f8ea0102780 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.144+0000 7f8ea4a2c700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f8e900692b0 0x7f8e9006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.144+0000 7f8ea4a2c700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f8e900692b0 0x7f8e9006b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8e940043d0 tx=0x7f8e94009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.148+0000 7f8e9d7fa700 1 -- 172.21.15.2:0/1809190908 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8e8c053260 con 0x7f8ea0102780 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.343+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8ea011a390 con 0x7f8ea0102780 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.343+0000 7f8e9d7fa700 1 -- 172.21.15.2:0/1809190908 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8e8c056880 con 0x7f8ea0102780 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f8e900692b0 msgr2=0x7f8e9006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f8e900692b0 0x7f8e9006b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8e940043d0 tx=0x7f8e94009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0102780 msgr2=0x7f8ea0103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0102780 0x7f8ea0103490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00b790 tx=0x7f8e8c009220 comp rx=0 tx=0).stop 2024-02-16T22:18:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 shutdown_connections 2024-02-16T22:18:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f8e900692b0 0x7f8e9006b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ea0101de0 0x7f8ea0102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 --2- 172.21.15.2:0/1809190908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ea0102780 0x7f8ea0103490 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 >> 172.21.15.2:0/1809190908 conn(0x7f8ea0071d50 msgr2=0x7f8ea0105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 shutdown_connections 2024-02-16T22:18:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:18.345+0000 7f8ea6c90700 1 -- 172.21.15.2:0/1809190908 wait complete. 2024-02-16T22:18:21.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:21 smithi002 ceph-mon[29154]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:21 smithi103 ceph-mon[36259]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:22.496 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724555 got 167503724556 for osd.5 2024-02-16T22:18:22.496 DEBUG:teuthology.parallel:result is None 2024-02-16T22:18:22.881 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705702 for osd.0 2024-02-16T22:18:22.882 DEBUG:teuthology.parallel:result is None 2024-02-16T22:18:23.240 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149716 got 111669149718 for osd.3 2024-02-16T22:18:23.240 DEBUG:teuthology.parallel:result is None 2024-02-16T22:18:23.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 -- 172.21.15.2:0/3440156958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10501038f0 msgr2=0x7f10500fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:23.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 --2- 172.21.15.2:0/3440156958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10501038f0 0x7f10500fcf60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1044004940 tx=0x7f1044009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:23.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 -- 172.21.15.2:0/3440156958 shutdown_connections 2024-02-16T22:18:23.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 --2- 172.21.15.2:0/3440156958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10501038f0 0x7f10500fcf60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:23.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 --2- 172.21.15.2:0/3440156958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1050102f50 0x7f1050103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:23.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 -- 172.21.15.2:0/3440156958 >> 172.21.15.2:0/3440156958 conn(0x7f10500f8ae0 msgr2=0x7f10500faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:23.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 -- 172.21.15.2:0/3440156958 shutdown_connections 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 -- 172.21.15.2:0/3440156958 wait complete. 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 Processor -- start 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 -- start start 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1050102f50 0x7f1050113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10501038f0 0x7f1050114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10501146e0 con 0x7f1050102f50 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.723+0000 7f105773f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105010d980 con 0x7f10501038f0 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f10554db700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1050102f50 0x7f1050113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f10554db700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1050102f50 0x7f1050113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50692/0 (socket says 172.21.15.2:50692) 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f10554db700 1 -- 172.21.15.2:0/1205576349 learned_addr learned my addr 172.21.15.2:0/1205576349 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f1054cda700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10501038f0 0x7f1050114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f10554db700 1 -- 172.21.15.2:0/1205576349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10501038f0 msgr2=0x7f1050114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:23.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f10554db700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10501038f0 0x7f1050114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:23.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f10554db700 1 -- 172.21.15.2:0/1205576349 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044005040 con 0x7f1050102f50 2024-02-16T22:18:23.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f10554db700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1050102f50 0x7f1050113ac0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f104000bbb0 tx=0x7f1040009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:23.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f104dffb700 1 -- 172.21.15.2:0/1205576349 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10400092e0 con 0x7f1050102f50 2024-02-16T22:18:23.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105010dc60 con 0x7f1050102f50 2024-02-16T22:18:23.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f104dffb700 1 -- 172.21.15.2:0/1205576349 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f104000cb20 con 0x7f1050102f50 2024-02-16T22:18:23.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f104dffb700 1 -- 172.21.15.2:0/1205576349 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1040015d70 con 0x7f1050102f50 2024-02-16T22:18:23.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.724+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105010e170 con 0x7f1050102f50 2024-02-16T22:18:23.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.725+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10500fc6e0 con 0x7f1050102f50 2024-02-16T22:18:23.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.726+0000 7f104dffb700 1 -- 172.21.15.2:0/1205576349 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f1040012070 con 0x7f1050102f50 2024-02-16T22:18:23.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.726+0000 7f104dffb700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f10380692a0 0x7f103806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:23.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.726+0000 7f104dffb700 1 -- 172.21.15.2:0/1205576349 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1040086b90 con 0x7f1050102f50 2024-02-16T22:18:23.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.726+0000 7f1054cda700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f10380692a0 0x7f103806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:23.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.727+0000 7f1054cda700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f10380692a0 0x7f103806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f104400b3b0 tx=0x7f1044009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:23.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.731+0000 7f104dffb700 1 -- 172.21.15.2:0/1205576349 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1040056c10 con 0x7f1050102f50 2024-02-16T22:18:23.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.923+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f105010f0a0 con 0x7f1050102f50 2024-02-16T22:18:23.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.923+0000 7f104dffb700 1 -- 172.21.15.2:0/1205576349 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f10400567a0 con 0x7f1050102f50 2024-02-16T22:18:23.925 INFO:teuthology.orchestra.run.smithi002.stdout:81604378650 2024-02-16T22:18:23.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f10380692a0 msgr2=0x7f103806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:23.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f10380692a0 0x7f103806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f104400b3b0 tx=0x7f1044009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:23.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1050102f50 msgr2=0x7f1050113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:23.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1050102f50 0x7f1050113ac0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f104000bbb0 tx=0x7f1040009840 comp rx=0 tx=0).stop 2024-02-16T22:18:23.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 shutdown_connections 2024-02-16T22:18:23.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f10380692a0 0x7f103806b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:23.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1050102f50 0x7f1050113ac0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:23.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 --2- 172.21.15.2:0/1205576349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10501038f0 0x7f1050114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:23.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 >> 172.21.15.2:0/1205576349 conn(0x7f10500f8ae0 msgr2=0x7f10500fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:23.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 shutdown_connections 2024-02-16T22:18:23.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:23.925+0000 7f105773f700 1 -- 172.21.15.2:0/1205576349 wait complete. 2024-02-16T22:18:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:23 smithi002 ceph-mon[29154]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:18:24.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:23 smithi103 ceph-mon[36259]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:24.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:18:24.655 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378650 for osd.2 2024-02-16T22:18:24.656 DEBUG:teuthology.parallel:result is None 2024-02-16T22:18:24.656 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-16T22:18:24.656 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph pg dump --format=json 2024-02-16T22:18:24.927 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:25.150 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:24 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1205576349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-16T22:18:25.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:24 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1205576349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-16T22:18:25.950 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:25 smithi002 ceph-mon[29154]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:26.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:25 smithi103 ceph-mon[36259]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.451+0000 7f43dda67700 1 -- 172.21.15.2:0/640018292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 msgr2=0x7f43d8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.451+0000 7f43dda67700 1 --2- 172.21.15.2:0/640018292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 0x7f43d8103320 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f43c8004940 tx=0x7f43c8009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.451+0000 7f43dda67700 1 -- 172.21.15.2:0/640018292 shutdown_connections 2024-02-16T22:18:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.451+0000 7f43dda67700 1 --2- 172.21.15.2:0/640018292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43d81038f0 0x7f43d80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.451+0000 7f43dda67700 1 --2- 172.21.15.2:0/640018292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 0x7f43d8103320 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.451+0000 7f43dda67700 1 -- 172.21.15.2:0/640018292 >> 172.21.15.2:0/640018292 conn(0x7f43d80f8ae0 msgr2=0x7f43d80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.452+0000 7f43dda67700 1 -- 172.21.15.2:0/640018292 shutdown_connections 2024-02-16T22:18:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.452+0000 7f43dda67700 1 -- 172.21.15.2:0/640018292 wait complete. 2024-02-16T22:18:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.452+0000 7f43dda67700 1 Processor -- start 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.452+0000 7f43dda67700 1 -- start start 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.452+0000 7f43dda67700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 0x7f43d81139d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.452+0000 7f43dda67700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43d81038f0 0x7f43d8113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43dda67700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d8114560 con 0x7f43d8102f50 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43dda67700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d810d890 con 0x7f43d81038f0 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43d77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 0x7f43d81139d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43d77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 0x7f43d81139d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50716/0 (socket says 172.21.15.2:50716) 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43d77fe700 1 -- 172.21.15.2:0/1963885971 learned_addr learned my addr 172.21.15.2:0/1963885971 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43d6ffd700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43d81038f0 0x7f43d8113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43d77fe700 1 -- 172.21.15.2:0/1963885971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43d81038f0 msgr2=0x7f43d8113f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43d77fe700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43d81038f0 0x7f43d8113f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43d77fe700 1 -- 172.21.15.2:0/1963885971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c8005040 con 0x7f43d8102f50 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43d6ffd700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43d81038f0 0x7f43d8113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-16T22:18:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43d77fe700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 0x7f43d81139d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009b30 tx=0x7f43c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43bffff700 1 -- 172.21.15.2:0/1963885971 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43c8021070 con 0x7f43d8102f50 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43bffff700 1 -- 172.21.15.2:0/1963885971 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43c8013e50 con 0x7f43d8102f50 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d810db10 con 0x7f43d8102f50 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43bffff700 1 -- 172.21.15.2:0/1963885971 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43c800e070 con 0x7f43d8102f50 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.453+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d810dff0 con 0x7f43d8102f50 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.454+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d80fc6e0 con 0x7f43d8102f50 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.455+0000 7f43bffff700 1 -- 172.21.15.2:0/1963885971 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f43c801cc40 con 0x7f43d8102f50 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.455+0000 7f43bffff700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f43c00692a0 0x7f43c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.455+0000 7f43d6ffd700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f43c00692a0 0x7f43c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.455+0000 7f43bffff700 1 -- 172.21.15.2:0/1963885971 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f43c80883d0 con 0x7f43d8102f50 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.455+0000 7f43d6ffd700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f43c00692a0 0x7f43c006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43cc004600 tx=0x7f43cc00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.460+0000 7f43bffff700 1 -- 172.21.15.2:0/1963885971 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f43c80584d0 con 0x7f43d8102f50 2024-02-16T22:18:26.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.648+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f43d810f020 con 0x7f43c00692a0 2024-02-16T22:18:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.649+0000 7f43bffff700 1 -- 172.21.15.2:0/1963885971 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28107 (secure 0 0 0) 0x7f43d810f020 con 0x7f43c00692a0 2024-02-16T22:18:26.651 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f43c00692a0 msgr2=0x7f43c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f43c00692a0 0x7f43c006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43cc004600 tx=0x7f43cc00b040 comp rx=0 tx=0).stop 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 msgr2=0x7f43d81139d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 0x7f43d81139d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009b30 tx=0x7f43c8009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 shutdown_connections 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f43c00692a0 0x7f43c006b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43d8102f50 0x7f43d81139d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 --2- 172.21.15.2:0/1963885971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43d81038f0 0x7f43d8113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 >> 172.21.15.2:0/1963885971 conn(0x7f43d80f8ae0 msgr2=0x7f43d80fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 shutdown_connections 2024-02-16T22:18:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:26.651+0000 7f43dda67700 1 -- 172.21.15.2:0/1963885971 wait complete. 2024-02-16T22:18:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-02-16T22:18:27.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:26 smithi002 ceph-mon[29154]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:27.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:26 smithi002 ceph-mon[29154]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-16T22:18:27.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:26 smithi103 ceph-mon[36259]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:27.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:26 smithi103 ceph-mon[36259]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-16T22:18:27.334 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-02-16T22:18:26.576409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002192"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-16T22:17:32.874758+0000","last_change":"2024-02-16T22:17:07.382881+0000","last_active":"2024-02-16T22:17:32.874758+0000","last_peered":"2024-02-16T22:17:32.874758+0000","last_clean":"2024-02-16T22:17:32.874758+0000","last_became_active":"2024-02-16T22:17:07.382618+0000","last_became_peered":"2024-02-16T22:17:07.382618+0000","last_unstale":"2024-02-16T22:17:32.874758+0000","last_undegraded":"2024-02-16T22:17:32.874758+0000","last_fullsized":"2024-02-16T22:17:32.874758+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-16T22:16:19.389621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-16T22:16:19.389621+0000","last_clean_scrub_stamp":"2024-02-16T22:16:19.389621+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-18T00:21:15.087652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.51700000000000002}]},{"osd":1,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.45500000000000002}]},{"osd":3,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 16 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.57999999999999996}]},{"osd":1,"last update":"Fri Feb 16 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.3390000000000004,"5min":4.3390000000000004,"15min":4.3390000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.68700000000000006}]},{"osd":2,"last update":"Fri Feb 16 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.3600000000000003,"5min":4.3600000000000003,"15min":4.3600000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.70999999999999996}]},{"osd":4,"last update":"Fri Feb 16 22:18:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.443},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 16 22:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.27500000000000002}]},{"osd":1,"last update":"Fri Feb 16 22:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.248}]},{"osd":3,"last update":"Fri Feb 16 22:17:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.41599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 16 22:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.32800000000000001,"15min":0.32100000000000001},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.28199999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Feb 16 22:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.50600000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39200000000000002,"15min":0.39800000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55500000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.44900000000000001}]},{"osd":3,"last update":"Fri Feb 16 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 16 22:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.51300000000000001,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.35499999999999998},"min":{"1min":0.251,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.56899999999999995,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Feb 16 22:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.39000000000000001,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.46300000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.317},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.41099999999999998,"15min":0.42199999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.45100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.36499999999999999}]},{"osd":3,"last update":"Fri Feb 16 22:17:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.68200000000000005}]},{"osd":4,"last update":"Fri Feb 16 22:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.62},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-16T22:18:27.336 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph pg dump --format=json 2024-02-16T22:18:27.599 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:29.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 -- 172.21.15.2:0/3988593724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 msgr2=0x7fb174103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 --2- 172.21.15.2:0/3988593724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 0x7fb174103320 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb164009ce0 tx=0x7fb164004d10 comp rx=0 tx=0).stop 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 -- 172.21.15.2:0/3988593724 shutdown_connections 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 --2- 172.21.15.2:0/3988593724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1741038f0 0x7fb1740fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 --2- 172.21.15.2:0/3988593724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 0x7fb174103320 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 -- 172.21.15.2:0/3988593724 >> 172.21.15.2:0/3988593724 conn(0x7fb1740f8ae0 msgr2=0x7fb1740faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 -- 172.21.15.2:0/3988593724 shutdown_connections 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 -- 172.21.15.2:0/3988593724 wait complete. 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 Processor -- start 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.268+0000 7fb178c15700 1 -- start start 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb178c15700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 0x7fb174070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb178c15700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1741038f0 0x7fb1740707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb178c15700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb174070e60 con 0x7fb174102f50 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb178c15700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb174070fd0 con 0x7fb1741038f0 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb17259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 0x7fb174070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb17259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 0x7fb174070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50746/0 (socket says 172.21.15.2:50746) 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb17259c700 1 -- 172.21.15.2:0/3567596266 learned_addr learned my addr 172.21.15.2:0/3567596266 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb17259c700 1 -- 172.21.15.2:0/3567596266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1741038f0 msgr2=0x7fb1740707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb171d9b700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1741038f0 0x7fb1740707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb17259c700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1741038f0 0x7fb1740707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb17259c700 1 -- 172.21.15.2:0/3567596266 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb164009940 con 0x7fb174102f50 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb171d9b700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1741038f0 0x7fb1740707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb17259c700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 0x7fb174070290 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb16400bf60 tx=0x7fb164004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb16affd700 1 -- 172.21.15.2:0/3567596266 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb16400e040 con 0x7fb174102f50 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb16affd700 1 -- 172.21.15.2:0/3567596266 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1640128e0 con 0x7fb174102f50 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb174119e80 con 0x7fb174102f50 2024-02-16T22:18:29.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb16affd700 1 -- 172.21.15.2:0/3567596266 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb16401c870 con 0x7fb174102f50 2024-02-16T22:18:29.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.269+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17411a340 con 0x7fb174102f50 2024-02-16T22:18:29.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.270+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1740fc6e0 con 0x7fb174102f50 2024-02-16T22:18:29.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.271+0000 7fb16affd700 1 -- 172.21.15.2:0/3567596266 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fb164012a50 con 0x7fb174102f50 2024-02-16T22:18:29.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.271+0000 7fb16affd700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb1600692a0 0x7fb16006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:29.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.271+0000 7fb171d9b700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb1600692a0 0x7fb16006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:29.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.271+0000 7fb16affd700 1 -- 172.21.15.2:0/3567596266 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb164088b00 con 0x7fb174102f50 2024-02-16T22:18:29.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.272+0000 7fb171d9b700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb1600692a0 0x7fb16006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb15c006fd0 tx=0x7fb15c008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:29.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.276+0000 7fb16affd700 1 -- 172.21.15.2:0/3567596266 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb164055170 con 0x7fb174102f50 2024-02-16T22:18:29.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.461+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb174071a40 con 0x7fb1600692a0 2024-02-16T22:18:29.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.463+0000 7fb16affd700 1 -- 172.21.15.2:0/3567596266 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28095 (secure 0 0 0) 0x7fb174071a40 con 0x7fb1600692a0 2024-02-16T22:18:29.465 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:18:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb1600692a0 msgr2=0x7fb16006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb1600692a0 0x7fb16006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb15c006fd0 tx=0x7fb15c008040 comp rx=0 tx=0).stop 2024-02-16T22:18:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 msgr2=0x7fb174070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 0x7fb174070290 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb16400bf60 tx=0x7fb164004d10 comp rx=0 tx=0).stop 2024-02-16T22:18:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 shutdown_connections 2024-02-16T22:18:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb1600692a0 0x7fb16006b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb174102f50 0x7fb174070290 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 --2- 172.21.15.2:0/3567596266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb1741038f0 0x7fb1740707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 >> 172.21.15.2:0/3567596266 conn(0x7fb1740f8ae0 msgr2=0x7fb1740fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 shutdown_connections 2024-02-16T22:18:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:29.465+0000 7fb178c15700 1 -- 172.21.15.2:0/3567596266 wait complete. 2024-02-16T22:18:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-02-16T22:18:29.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:29 smithi002 ceph-mon[29154]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:30.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:29 smithi103 ceph-mon[36259]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:30.099 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-02-16T22:18:28.576835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002138"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-16T22:17:32.874758+0000","last_change":"2024-02-16T22:17:07.382881+0000","last_active":"2024-02-16T22:17:32.874758+0000","last_peered":"2024-02-16T22:17:32.874758+0000","last_clean":"2024-02-16T22:17:32.874758+0000","last_became_active":"2024-02-16T22:17:07.382618+0000","last_became_peered":"2024-02-16T22:17:07.382618+0000","last_unstale":"2024-02-16T22:17:32.874758+0000","last_undegraded":"2024-02-16T22:17:32.874758+0000","last_fullsized":"2024-02-16T22:17:32.874758+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-16T22:16:19.389621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-16T22:16:19.389621+0000","last_clean_scrub_stamp":"2024-02-16T22:16:19.389621+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-18T00:21:15.087652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.44700000000000001}]},{"osd":1,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.503}]},{"osd":2,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.39300000000000002}]},{"osd":3,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.36799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 16 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.57999999999999996}]},{"osd":1,"last update":"Fri Feb 16 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":4.3390000000000004,"5min":4.3390000000000004,"15min":4.3390000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.68700000000000006}]},{"osd":2,"last update":"Fri Feb 16 22:17:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.3600000000000003,"5min":4.3600000000000003,"15min":4.3600000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.70999999999999996}]},{"osd":4,"last update":"Fri Feb 16 22:18:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.443},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.48799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 16 22:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.27500000000000002}]},{"osd":1,"last update":"Fri Feb 16 22:17:23 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.248}]},{"osd":3,"last update":"Fri Feb 16 22:17:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.41599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 16 22:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.32800000000000001,"15min":0.32100000000000001},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.28199999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Feb 16 22:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.38500000000000001,"15min":0.38900000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.50600000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39200000000000002,"15min":0.39800000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.55500000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.44900000000000001}]},{"osd":3,"last update":"Fri Feb 16 22:17:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Fri Feb 16 22:18:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 16 22:17:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.51300000000000001,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.35499999999999998},"min":{"1min":0.251,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.56899999999999995,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Feb 16 22:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.39000000000000001,"15min":0.40000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.46300000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.317},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.41099999999999998,"15min":0.42199999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.45100000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.36499999999999999}]},{"osd":3,"last update":"Fri Feb 16 22:17:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.68200000000000005}]},{"osd":4,"last update":"Fri Feb 16 22:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.62},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-16T22:18:30.100 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-16T22:18:30.100 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-16T22:18:30.100 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-16T22:18:30.100 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph health --format=json 2024-02-16T22:18:30.372 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:31.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:30 smithi002 ceph-mon[29154]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-16T22:18:31.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:30 smithi103 ceph-mon[36259]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-16T22:18:32.138 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:31 smithi002 ceph-mon[29154]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 -- 172.21.15.2:0/2925708519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 msgr2=0x7fa9a0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 --2- 172.21.15.2:0/2925708519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 0x7fa9a0103320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa990004940 tx=0x7fa990009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 -- 172.21.15.2:0/2925708519 shutdown_connections 2024-02-16T22:18:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 --2- 172.21.15.2:0/2925708519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a01038f0 0x7fa9a00fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 --2- 172.21.15.2:0/2925708519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 0x7fa9a0103320 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 -- 172.21.15.2:0/2925708519 >> 172.21.15.2:0/2925708519 conn(0x7fa9a00f8ae0 msgr2=0x7fa9a00faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 -- 172.21.15.2:0/2925708519 shutdown_connections 2024-02-16T22:18:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 -- 172.21.15.2:0/2925708519 wait complete. 2024-02-16T22:18:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 Processor -- start 2024-02-16T22:18:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.164+0000 7fa9a69c0700 1 -- start start 2024-02-16T22:18:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa9a69c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 0x7fa9a0113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa9a69c0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a01038f0 0x7fa9a0113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa9a69c0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01145f0 con 0x7fa9a0102f50 2024-02-16T22:18:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa9a69c0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a010d920 con 0x7fa9a01038f0 2024-02-16T22:18:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 0x7fa9a0113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 0x7fa9a0113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38852/0 (socket says 172.21.15.2:38852) 2024-02-16T22:18:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99ffff700 1 -- 172.21.15.2:0/1747340150 learned_addr learned my addr 172.21.15.2:0/1747340150 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99f7fe700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a01038f0 0x7fa9a0113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99ffff700 1 -- 172.21.15.2:0/1747340150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a01038f0 msgr2=0x7fa9a0113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99ffff700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a01038f0 0x7fa9a0113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99ffff700 1 -- 172.21.15.2:0/1747340150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa990005040 con 0x7fa9a0102f50 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99ffff700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 0x7fa9a0113a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa99000b3b0 tx=0x7fa990009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99cff9700 1 -- 172.21.15.2:0/1747340150 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa99000e040 con 0x7fa9a0102f50 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99cff9700 1 -- 172.21.15.2:0/1747340150 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa99000bbf0 con 0x7fa9a0102f50 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a010dba0 con 0x7fa9a0102f50 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa99cff9700 1 -- 172.21.15.2:0/1747340150 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa99001c3f0 con 0x7fa9a0102f50 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.165+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a010e0e0 con 0x7fa9a0102f50 2024-02-16T22:18:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.166+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a0100260 con 0x7fa9a0102f50 2024-02-16T22:18:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.167+0000 7fa99cff9700 1 -- 172.21.15.2:0/1747340150 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa99000bd60 con 0x7fa9a0102f50 2024-02-16T22:18:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.167+0000 7fa99cff9700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9880692a0 0x7fa98806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.167+0000 7fa99cff9700 1 -- 172.21.15.2:0/1747340150 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa9900881c0 con 0x7fa9a0102f50 2024-02-16T22:18:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.167+0000 7fa99f7fe700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9880692a0 0x7fa98806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.168+0000 7fa99f7fe700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9880692a0 0x7fa98806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa9a010f290 tx=0x7fa99400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:32.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.172+0000 7fa99cff9700 1 -- 172.21.15.2:0/1747340150 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa9900582c0 con 0x7fa9a0102f50 2024-02-16T22:18:32.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:31 smithi103 ceph-mon[36259]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.406+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa9a010a050 con 0x7fa9a0102f50 2024-02-16T22:18:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.407+0000 7fa99cff9700 1 -- 172.21.15.2:0/1747340150 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa990057e50 con 0x7fa9a0102f50 2024-02-16T22:18:32.409 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:18:32.409 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-16T22:18:32.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9880692a0 msgr2=0x7fa98806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:32.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9880692a0 0x7fa98806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa9a010f290 tx=0x7fa99400b040 comp rx=0 tx=0).stop 2024-02-16T22:18:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 msgr2=0x7fa9a0113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 0x7fa9a0113a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa99000b3b0 tx=0x7fa990009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 shutdown_connections 2024-02-16T22:18:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9880692a0 0x7fa98806b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9a0102f50 0x7fa9a0113a60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 --2- 172.21.15.2:0/1747340150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9a01038f0 0x7fa9a0113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 >> 172.21.15.2:0/1747340150 conn(0x7fa9a00f8ae0 msgr2=0x7fa9a00fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 shutdown_connections 2024-02-16T22:18:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:32.409+0000 7fa9a69c0700 1 -- 172.21.15.2:0/1747340150 wait complete. 2024-02-16T22:18:33.125 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:32 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1747340150' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-16T22:18:33.153 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-16T22:18:33.154 INFO:tasks.cephadm:Setup complete, yielding 2024-02-16T22:18:33.154 INFO:teuthology.run_tasks:Running task print... 2024-02-16T22:18:33.163 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-02-16T22:18:33.163 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-16T22:18:33.200 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:18:33.200 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-16T22:18:33.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:32 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1747340150' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-16T22:18:33.464 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:34.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:33 smithi002 ceph-mon[29154]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:34.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:33 smithi103 ceph-mon[36259]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:35.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.015+0000 7f1af59c2700 1 -- 172.21.15.2:0/2520787830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 msgr2=0x7f1af00fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.015+0000 7f1af59c2700 1 --2- 172.21.15.2:0/2520787830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 0x7f1af00fd320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4004940 tx=0x7f1ae4009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 -- 172.21.15.2:0/2520787830 shutdown_connections 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 --2- 172.21.15.2:0/2520787830 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1af00fe0e0 0x7f1af00fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 --2- 172.21.15.2:0/2520787830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 0x7f1af00fd320 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 -- 172.21.15.2:0/2520787830 >> 172.21.15.2:0/2520787830 conn(0x7f1af00f8620 msgr2=0x7f1af00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 -- 172.21.15.2:0/2520787830 shutdown_connections 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 -- 172.21.15.2:0/2520787830 wait complete. 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 Processor -- start 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 -- start start 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 0x7f1af0071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1af00fe0e0 0x7f1af00702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af00708e0 con 0x7f1af00fcef0 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.016+0000 7f1af59c2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af0070a50 con 0x7f1af00fe0e0 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1aeeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 0x7f1af0071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:35.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1aeeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 0x7f1af0071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38868/0 (socket says 172.21.15.2:38868) 2024-02-16T22:18:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1aeeffd700 1 -- 172.21.15.2:0/2064285848 learned_addr learned my addr 172.21.15.2:0/2064285848 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1aee7fc700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1af00fe0e0 0x7f1af00702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1aeeffd700 1 -- 172.21.15.2:0/2064285848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1af00fe0e0 msgr2=0x7f1af00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1aeeffd700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1af00fe0e0 0x7f1af00702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1aeeffd700 1 -- 172.21.15.2:0/2064285848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae4005040 con 0x7f1af00fcef0 2024-02-16T22:18:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1aeeffd700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 0x7f1af0071e60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4000c00 tx=0x7f1ae4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1ad77fe700 1 -- 172.21.15.2:0/2064285848 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae400e070 con 0x7f1af00fcef0 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af0070cd0 con 0x7f1af00fcef0 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1ad77fe700 1 -- 172.21.15.2:0/2064285848 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ae401c400 con 0x7f1af00fcef0 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1ad77fe700 1 -- 172.21.15.2:0/2064285848 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae400b670 con 0x7f1af00fcef0 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.017+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af0115920 con 0x7f1af00fcef0 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.018+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af004bed0 con 0x7f1af00fcef0 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.019+0000 7f1ad77fe700 1 -- 172.21.15.2:0/2064285848 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f1ae401c9e0 con 0x7f1af00fcef0 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.019+0000 7f1ad77fe700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1ad80692a0 0x7f1ad806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.019+0000 7f1ad77fe700 1 -- 172.21.15.2:0/2064285848 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1ae40884c0 con 0x7f1af00fcef0 2024-02-16T22:18:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.019+0000 7f1aee7fc700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1ad80692a0 0x7f1ad806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.020+0000 7f1aee7fc700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1ad80692a0 0x7f1ad806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009b80 tx=0x7f1ae000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:35.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.024+0000 7f1ad77fe700 1 -- 172.21.15.2:0/2064285848 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1ae40585c0 con 0x7f1af00fcef0 2024-02-16T22:18:35.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.214+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f1af0105a30 con 0x7f1af00fcef0 2024-02-16T22:18:35.216 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:34 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:18:35.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.220+0000 7f1ad77fe700 1 -- 172.21.15.2:0/2064285848 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f1ae4058150 con 0x7f1af00fcef0 2024-02-16T22:18:35.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1ad80692a0 msgr2=0x7f1ad806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:35.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1ad80692a0 0x7f1ad806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009b80 tx=0x7f1ae000b040 comp rx=0 tx=0).stop 2024-02-16T22:18:35.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 msgr2=0x7f1af0071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:35.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 0x7f1af0071e60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4000c00 tx=0x7f1ae4009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:35.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 shutdown_connections 2024-02-16T22:18:35.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f1ad80692a0 0x7f1ad806b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:35.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1af00fcef0 0x7f1af0071e60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:35.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 --2- 172.21.15.2:0/2064285848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1af00fe0e0 0x7f1af00702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:35.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 >> 172.21.15.2:0/2064285848 conn(0x7f1af00f8620 msgr2=0x7f1af01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:35.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 shutdown_connections 2024-02-16T22:18:35.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:35.222+0000 7f1af59c2700 1 -- 172.21.15.2:0/2064285848 wait complete. 2024-02-16T22:18:35.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:34 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:18:35.895 INFO:teuthology.run_tasks:Running task print... 2024-02-16T22:18:35.902 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-16T22:18:35.903 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-16T22:18:35.941 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:18:35.941 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph orch status' 2024-02-16T22:18:36.203 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:36.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:35 smithi002 ceph-mon[29154]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:36.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:35 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:36.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:35 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:18:36.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:35 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:36.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:35 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2064285848' entity='client.admin' 2024-02-16T22:18:36.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:35 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:18:36.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:35 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:36.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:35 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:18:36.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:35 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:36.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:35 smithi103 ceph-mon[36259]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:36.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:35 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:36.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:35 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:18:36.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:35 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:36.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:35 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2064285848' entity='client.admin' 2024-02-16T22:18:36.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:35 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:18:36.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:35 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:36.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:35 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:18:36.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:35 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:37.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:36 smithi002 ceph-mon[29154]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:37.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:36 smithi103 ceph-mon[36259]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:37.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.808+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2491841075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 msgr2=0x7f3e7c0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:37.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.808+0000 7f3e83cb7700 1 --2- 172.21.15.2:0/2491841075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 0x7f3e7c0fc250 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3e74004940 tx=0x7f3e74009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:37.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.808+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2491841075 shutdown_connections 2024-02-16T22:18:37.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.808+0000 7f3e83cb7700 1 --2- 172.21.15.2:0/2491841075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e7c0fcfb0 0x7f3e7c0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.808+0000 7f3e83cb7700 1 --2- 172.21.15.2:0/2491841075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 0x7f3e7c0fc250 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.808+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2491841075 >> 172.21.15.2:0/2491841075 conn(0x7f3e7c0f7500 msgr2=0x7f3e7c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.808+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2491841075 shutdown_connections 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.808+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2491841075 wait complete. 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.808+0000 7f3e83cb7700 1 Processor -- start 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e83cb7700 1 -- start start 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e83cb7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 0x7f3e7c10c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e83cb7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e7c0fcfb0 0x7f3e7c10cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e83cb7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c10d380 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e83cb7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c10dc20 con 0x7f3e7c0fcfb0 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e81a53700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 0x7f3e7c10c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e81a53700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 0x7f3e7c10c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38892/0 (socket says 172.21.15.2:38892) 2024-02-16T22:18:37.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e81a53700 1 -- 172.21.15.2:0/2195875931 learned_addr learned my addr 172.21.15.2:0/2195875931 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e81252700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e7c0fcfb0 0x7f3e7c10cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e81a53700 1 -- 172.21.15.2:0/2195875931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e7c0fcfb0 msgr2=0x7f3e7c10cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e81a53700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e7c0fcfb0 0x7f3e7c10cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e81a53700 1 -- 172.21.15.2:0/2195875931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74005040 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e81a53700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 0x7f3e7c10c840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e740093b0 tx=0x7f3e74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e7a7fc700 1 -- 172.21.15.2:0/2195875931 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7400e040 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e7a7fc700 1 -- 172.21.15.2:0/2195875931 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e7400bcc0 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7c10dea0 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.809+0000 7f3e7a7fc700 1 -- 172.21.15.2:0/2195875931 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7401c630 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.810+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7c10e3e0 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.810+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7c04bed0 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.811+0000 7f3e7a7fc700 1 -- 172.21.15.2:0/2195875931 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3e74013980 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.811+0000 7f3e7a7fc700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e640692a0 0x7f3e6406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.811+0000 7f3e7a7fc700 1 -- 172.21.15.2:0/2195875931 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3e740881e0 con 0x7f3e7c0fbe40 2024-02-16T22:18:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.811+0000 7f3e81252700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e640692a0 0x7f3e6406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.812+0000 7f3e81252700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e640692a0 0x7f3e6406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3e70004f40 tx=0x7f3e7000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:37.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:37.816+0000 7f3e7a7fc700 1 -- 172.21.15.2:0/2195875931 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3e740582e0 con 0x7f3e7c0fbe40 2024-02-16T22:18:38.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.020+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e7c101990 con 0x7f3e640692a0 2024-02-16T22:18:38.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.021+0000 7f3e7a7fc700 1 -- 172.21.15.2:0/2195875931 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3e7c101990 con 0x7f3e640692a0 2024-02-16T22:18:38.023 INFO:teuthology.orchestra.run.smithi002.stdout:Backend: cephadm 2024-02-16T22:18:38.023 INFO:teuthology.orchestra.run.smithi002.stdout:Available: Yes 2024-02-16T22:18:38.023 INFO:teuthology.orchestra.run.smithi002.stdout:Paused: No 2024-02-16T22:18:38.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.023+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e640692a0 msgr2=0x7f3e6406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:38.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.023+0000 7f3e83cb7700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e640692a0 0x7f3e6406b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3e70004f40 tx=0x7f3e7000b040 comp rx=0 tx=0).stop 2024-02-16T22:18:38.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.023+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 msgr2=0x7f3e7c10c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:38.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.023+0000 7f3e83cb7700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 0x7f3e7c10c840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e740093b0 tx=0x7f3e74009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:38.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.023+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 shutdown_connections 2024-02-16T22:18:38.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.023+0000 7f3e83cb7700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3e640692a0 0x7f3e6406b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:38.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.023+0000 7f3e83cb7700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e7c0fbe40 0x7f3e7c10c840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:38.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.023+0000 7f3e83cb7700 1 --2- 172.21.15.2:0/2195875931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e7c0fcfb0 0x7f3e7c10cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:38.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.023+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 >> 172.21.15.2:0/2195875931 conn(0x7f3e7c0f7500 msgr2=0x7f3e7c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:38.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.024+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 shutdown_connections 2024-02-16T22:18:38.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:38.024+0000 7f3e83cb7700 1 -- 172.21.15.2:0/2195875931 wait complete. 2024-02-16T22:18:38.676 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-16T22:18:38.936 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:38.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:18:39.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:18:39.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:39 smithi002 ceph-mon[29154]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:39.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:39 smithi002 ceph-mon[29154]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:40.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:39 smithi103 ceph-mon[36259]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:40.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:39 smithi103 ceph-mon[36259]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:40.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/3087101759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fcfb0 msgr2=0x7fb6a40fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:40.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 --2- 172.21.15.2:0/3087101759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fcfb0 0x7fb6a40fd420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb698004940 tx=0x7fb698009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:40.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/3087101759 shutdown_connections 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 --2- 172.21.15.2:0/3087101759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fcfb0 0x7fb6a40fd420 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 --2- 172.21.15.2:0/3087101759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6a40fbe40 0x7fb6a40fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/3087101759 >> 172.21.15.2:0/3087101759 conn(0x7fb6a40f7520 msgr2=0x7fb6a40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/3087101759 shutdown_connections 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/3087101759 wait complete. 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 Processor -- start 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 -- start start 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fbe40 0x7fb6a4110600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6a40fcfb0 0x7fb6a4112b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4066d50 con 0x7fb6a40fbe40 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.536+0000 7fb6a8fd8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4066e90 con 0x7fb6a40fcfb0 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fbe40 0x7fb6a4110600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fbe40 0x7fb6a4110600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42964/0 (socket says 172.21.15.2:42964) 2024-02-16T22:18:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a259c700 1 -- 172.21.15.2:0/50052589 learned_addr learned my addr 172.21.15.2:0/50052589 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a1d9b700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6a40fcfb0 0x7fb6a4112b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a259c700 1 -- 172.21.15.2:0/50052589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6a40fcfb0 msgr2=0x7fb6a4112b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a259c700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6a40fcfb0 0x7fb6a4112b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a259c700 1 -- 172.21.15.2:0/50052589 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb698005040 con 0x7fb6a40fbe40 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a259c700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fbe40 0x7fb6a4110600 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb694002a80 tx=0x7fb6940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb692ffd700 1 -- 172.21.15.2:0/50052589 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb694007cc0 con 0x7fb6a40fbe40 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb692ffd700 1 -- 172.21.15.2:0/50052589 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb69400f040 con 0x7fb6a40fbe40 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a41131b0 con 0x7fb6a40fbe40 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb692ffd700 1 -- 172.21.15.2:0/50052589 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb69400daf0 con 0x7fb6a40fbe40 2024-02-16T22:18:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.537+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a41136f0 con 0x7fb6a40fbe40 2024-02-16T22:18:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.538+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a404bed0 con 0x7fb6a40fbe40 2024-02-16T22:18:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.539+0000 7fb692ffd700 1 -- 172.21.15.2:0/50052589 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fb69400dc50 con 0x7fb6a40fbe40 2024-02-16T22:18:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.539+0000 7fb692ffd700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb68c0692a0 0x7fb68c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.539+0000 7fb692ffd700 1 -- 172.21.15.2:0/50052589 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb694086b40 con 0x7fb6a40fbe40 2024-02-16T22:18:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.539+0000 7fb6a1d9b700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb68c0692a0 0x7fb68c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.539+0000 7fb6a1d9b700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb68c0692a0 0x7fb68c06b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb69800b3b0 tx=0x7fb698009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:40.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.543+0000 7fb692ffd700 1 -- 172.21.15.2:0/50052589 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb694053130 con 0x7fb6a40fbe40 2024-02-16T22:18:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.742+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6a4101990 con 0x7fb68c0692a0 2024-02-16T22:18:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.750+0000 7fb692ffd700 1 -- 172.21.15.2:0/50052589 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7fb6a4101990 con 0x7fb68c0692a0 2024-02-16T22:18:40.752 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-16T22:18:40.752 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.smithi002 smithi002 *:9093,9094 running (3m) 2m ago 5m 26.1M - 0.25.0 c8568f914cd2 2b7780bcc981 2024-02-16T22:18:40.752 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi002 smithi002 running (5m) 2m ago 5m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 c1d597f23cdd 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi103 smithi103 running (4m) 70s ago 4m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 db2a59e09a51 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.smithi002 smithi002 *:3000 running (3m) 2m ago 4m 78.6M - 9.4.7 954c08fa6188 14099b699363 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi002.resmjv smithi002 *:9283 running (6m) 2m ago 6m 472M - 17.2.7-513-gb88fae66 870a0944ccb8 f9b132a0aa69 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi103.ydydbo smithi103 *:8443,9283 running (4m) 70s ago 4m 432M - 17.2.7-513-gb88fae66 870a0944ccb8 463897d4efc5 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi002 smithi002 running (6m) 2m ago 6m 45.4M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 cc6b0b4bb587 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi103 smithi103 running (4m) 70s ago 4m 38.7M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 3d198bee2a92 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi002 smithi002 *:9100 running (5m) 2m ago 5m 15.6M - 1.5.0 0da6a335fe13 e6146a57c445 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi103 smithi103 *:9100 running (4m) 70s ago 4m 16.9M - 1.5.0 0da6a335fe13 0b21b215b785 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (3m) 2m ago 3m 42.3M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 a3f19d068baf 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (2m) 2m ago 2m 39.4M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 58697cab97b0 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (2m) 2m ago 2m 42.1M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 634798c2dbad 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi103 running (2m) 70s ago 2m 38.5M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 a5332c698672 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi103 running (100s) 70s ago 100s 36.9M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 fa2a12f4b710 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi103 running (75s) 70s ago 74s 11.0M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 175f4eefd1f6 2024-02-16T22:18:40.753 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.smithi002 smithi002 *:9095 running (3m) 2m ago 4m 35.9M - 2.43.0 a07b618ecd1d d03f44725127 2024-02-16T22:18:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb68c0692a0 msgr2=0x7fb68c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb68c0692a0 0x7fb68c06b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb69800b3b0 tx=0x7fb698009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fbe40 msgr2=0x7fb6a4110600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fbe40 0x7fb6a4110600 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb694002a80 tx=0x7fb6940046b0 comp rx=0 tx=0).stop 2024-02-16T22:18:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 shutdown_connections 2024-02-16T22:18:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fb68c0692a0 0x7fb68c06b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6a40fbe40 0x7fb6a4110600 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 --2- 172.21.15.2:0/50052589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6a40fcfb0 0x7fb6a4112b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 >> 172.21.15.2:0/50052589 conn(0x7fb6a40f7520 msgr2=0x7fb6a4100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 shutdown_connections 2024-02-16T22:18:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:40.752+0000 7fb6a8fd8700 1 -- 172.21.15.2:0/50052589 wait complete. 2024-02-16T22:18:41.367 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-16T22:18:41.632 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:42.083 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:41 smithi002 ceph-mon[29154]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:42.084 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:41 smithi002 ceph-mon[29154]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:42.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:41 smithi103 ceph-mon[36259]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:42.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:41 smithi103 ceph-mon[36259]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.177+0000 7fdfe98f7700 1 -- 172.21.15.2:0/3116459483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 msgr2=0x7fdfe40fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.177+0000 7fdfe98f7700 1 --2- 172.21.15.2:0/3116459483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 0x7fdfe40fe550 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8004940 tx=0x7fdfd8009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.177+0000 7fdfe98f7700 1 -- 172.21.15.2:0/3116459483 shutdown_connections 2024-02-16T22:18:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.177+0000 7fdfe98f7700 1 --2- 172.21.15.2:0/3116459483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 0x7fdfe40fe550 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.177+0000 7fdfe98f7700 1 --2- 172.21.15.2:0/3116459483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfe40fcef0 0x7fdfe40fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.177+0000 7fdfe98f7700 1 -- 172.21.15.2:0/3116459483 >> 172.21.15.2:0/3116459483 conn(0x7fdfe40f8620 msgr2=0x7fdfe40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:43.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.177+0000 7fdfe98f7700 1 -- 172.21.15.2:0/3116459483 shutdown_connections 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.178+0000 7fdfe98f7700 1 -- 172.21.15.2:0/3116459483 wait complete. 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.178+0000 7fdfe98f7700 1 Processor -- start 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.178+0000 7fdfe98f7700 1 -- start start 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.178+0000 7fdfe98f7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfe40fcef0 0x7fdfe4071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.178+0000 7fdfe98f7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 0x7fdfe40702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.178+0000 7fdfe98f7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe40708e0 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.178+0000 7fdfe98f7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe4070a50 con 0x7fdfe40fcef0 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.178+0000 7fdfe27fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 0x7fdfe40702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfe27fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 0x7fdfe40702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42986/0 (socket says 172.21.15.2:42986) 2024-02-16T22:18:43.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfe27fc700 1 -- 172.21.15.2:0/4160094107 learned_addr learned my addr 172.21.15.2:0/4160094107 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfe2ffd700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfe40fcef0 0x7fdfe4071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfe27fc700 1 -- 172.21.15.2:0/4160094107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfe40fcef0 msgr2=0x7fdfe4071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfe27fc700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfe40fcef0 0x7fdfe4071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfe27fc700 1 -- 172.21.15.2:0/4160094107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd8005040 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfe27fc700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 0x7fdfe40702e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80093b0 tx=0x7fdfd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfcb7fe700 1 -- 172.21.15.2:0/4160094107 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd800e040 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe4070cd0 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfcb7fe700 1 -- 172.21.15.2:0/4160094107 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfd800bcc0 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfcb7fe700 1 -- 172.21.15.2:0/4160094107 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd801c630 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.179+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe4115990 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.180+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe404bed0 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.181+0000 7fdfcb7fe700 1 -- 172.21.15.2:0/4160094107 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fdfd8013980 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.181+0000 7fdfcb7fe700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdfcc0692a0 0x7fdfcc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.181+0000 7fdfcb7fe700 1 -- 172.21.15.2:0/4160094107 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdfd80881e0 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.181+0000 7fdfe2ffd700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdfcc0692a0 0x7fdfcc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.181+0000 7fdfe2ffd700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdfcc0692a0 0x7fdfcc06b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4071bf0 tx=0x7fdfd4009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:43.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.186+0000 7fdfcb7fe700 1 -- 172.21.15.2:0/4160094107 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdfd80582e0 con 0x7fdfe40fe0e0 2024-02-16T22:18:43.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.384+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfe4102ac0 con 0x7fdfcc0692a0 2024-02-16T22:18:43.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.388+0000 7fdfcb7fe700 1 -- 172.21.15.2:0/4160094107 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1161 (secure 0 0 0) 0x7fdfe4102ac0 con 0x7fdfcc0692a0 2024-02-16T22:18:43.390 INFO:teuthology.orchestra.run.smithi002.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-16T22:18:43.390 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-02-16T22:18:43.391 INFO:teuthology.orchestra.run.smithi002.stdout:crash 2/2 2m ago 6m * 2024-02-16T22:18:43.391 INFO:teuthology.orchestra.run.smithi002.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-02-16T22:18:43.391 INFO:teuthology.orchestra.run.smithi002.stdout:mgr 2/2 2m ago 6m count:2 2024-02-16T22:18:43.391 INFO:teuthology.orchestra.run.smithi002.stdout:mon 2/2 2m ago 5m smithi002:172.21.15.2=smithi002;smithi103:172.21.15.103=smithi103;count:2 2024-02-16T22:18:43.391 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-02-16T22:18:43.391 INFO:teuthology.orchestra.run.smithi002.stdout:osd 6 2m ago - 2024-02-16T22:18:43.391 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-02-16T22:18:43.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdfcc0692a0 msgr2=0x7fdfcc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdfcc0692a0 0x7fdfcc06b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4071bf0 tx=0x7fdfd4009040 comp rx=0 tx=0).stop 2024-02-16T22:18:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 msgr2=0x7fdfe40702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 0x7fdfe40702e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80093b0 tx=0x7fdfd8009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 shutdown_connections 2024-02-16T22:18:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fdfcc0692a0 0x7fdfcc06b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfe40fcef0 0x7fdfe4071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 --2- 172.21.15.2:0/4160094107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfe40fe0e0 0x7fdfe40702e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 >> 172.21.15.2:0/4160094107 conn(0x7fdfe40f8620 msgr2=0x7fdfe41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 shutdown_connections 2024-02-16T22:18:43.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:43.390+0000 7fdfe98f7700 1 -- 172.21.15.2:0/4160094107 wait complete. 2024-02-16T22:18:43.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:43 smithi002 ceph-mon[29154]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:44.019 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-16T22:18:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:44.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:43 smithi103 ceph-mon[36259]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:44.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:44 smithi002 ceph-mon[29154]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:45.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:44 smithi103 ceph-mon[36259]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:45.904 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:45 smithi002 ceph-mon[29154]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:46.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.003+0000 7f1671481700 1 -- 172.21.15.2:0/3020241598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fe640 msgr2=0x7f166c0fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:46.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.003+0000 7f1671481700 1 --2- 172.21.15.2:0/3020241598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fe640 0x7f166c0fea50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f165c004940 tx=0x7f165c009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:46.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.003+0000 7f1671481700 1 -- 172.21.15.2:0/3020241598 shutdown_connections 2024-02-16T22:18:46.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.003+0000 7f1671481700 1 --2- 172.21.15.2:0/3020241598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c0fef90 0x7f166c0fb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:46.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.003+0000 7f1671481700 1 --2- 172.21.15.2:0/3020241598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fe640 0x7f166c0fea50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:46.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.003+0000 7f1671481700 1 -- 172.21.15.2:0/3020241598 >> 172.21.15.2:0/3020241598 conn(0x7f166c0f6d20 msgr2=0x7f166c0f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:46.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.003+0000 7f1671481700 1 -- 172.21.15.2:0/3020241598 shutdown_connections 2024-02-16T22:18:46.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.003+0000 7f1671481700 1 -- 172.21.15.2:0/3020241598 wait complete. 2024-02-16T22:18:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f1671481700 1 Processor -- start 2024-02-16T22:18:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f1671481700 1 -- start start 2024-02-16T22:18:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f1671481700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c0fe640 0x7f166c10dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f1671481700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fef90 0x7f166c10e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f1671481700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c10e890 con 0x7f166c0fef90 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f1671481700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c10f020 con 0x7f166c0fe640 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f166a7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fef90 0x7f166c10e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f166a7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fef90 0x7f166c10e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42998/0 (socket says 172.21.15.2:42998) 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f166a7fc700 1 -- 172.21.15.2:0/3069939959 learned_addr learned my addr 172.21.15.2:0/3069939959 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f166affd700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c0fe640 0x7f166c10dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f166a7fc700 1 -- 172.21.15.2:0/3069939959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c0fe640 msgr2=0x7f166c10dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f166a7fc700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c0fe640 0x7f166c10dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.004+0000 7f166a7fc700 1 -- 172.21.15.2:0/3069939959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c005040 con 0x7f166c0fef90 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.005+0000 7f166a7fc700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fef90 0x7f166c10e290 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f166000c830 tx=0x7f1660004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.005+0000 7f16537fe700 1 -- 172.21.15.2:0/3069939959 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f166000a760 con 0x7f166c0fef90 2024-02-16T22:18:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.005+0000 7f16537fe700 1 -- 172.21.15.2:0/3069939959 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f166000ad90 con 0x7f166c0fef90 2024-02-16T22:18:46.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.005+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c10f300 con 0x7f166c0fef90 2024-02-16T22:18:46.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.005+0000 7f16537fe700 1 -- 172.21.15.2:0/3069939959 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16600113a0 con 0x7f166c0fef90 2024-02-16T22:18:46.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.005+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c10f840 con 0x7f166c0fef90 2024-02-16T22:18:46.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.006+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c107a90 con 0x7f166c0fef90 2024-02-16T22:18:46.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.006+0000 7f16537fe700 1 -- 172.21.15.2:0/3069939959 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f1660012400 con 0x7f166c0fef90 2024-02-16T22:18:46.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.007+0000 7f16537fe700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f16540692a0 0x7f165406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:46.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.007+0000 7f16537fe700 1 -- 172.21.15.2:0/3069939959 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1660086ef0 con 0x7f166c0fef90 2024-02-16T22:18:46.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.007+0000 7f166affd700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f16540692a0 0x7f165406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:46.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.007+0000 7f166affd700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f16540692a0 0x7f165406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f165c004f90 tx=0x7f165c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:46.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.011+0000 7f16537fe700 1 -- 172.21.15.2:0/3069939959 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1660056ff0 con 0x7f166c0fef90 2024-02-16T22:18:46.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.209+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f166c105c10 con 0x7f16540692a0 2024-02-16T22:18:46.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.211+0000 7f16537fe700 1 -- 172.21.15.2:0/3069939959 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f166c105c10 con 0x7f16540692a0 2024-02-16T22:18:46.213 INFO:teuthology.orchestra.run.smithi002.stdout:HOST ADDR LABELS STATUS 2024-02-16T22:18:46.213 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 172.21.15.2 2024-02-16T22:18:46.213 INFO:teuthology.orchestra.run.smithi002.stdout:smithi103 172.21.15.103 2024-02-16T22:18:46.213 INFO:teuthology.orchestra.run.smithi002.stdout:2 hosts in cluster 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f16540692a0 msgr2=0x7f165406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f16540692a0 0x7f165406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f165c004f90 tx=0x7f165c009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fef90 msgr2=0x7f166c10e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fef90 0x7f166c10e290 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f166000c830 tx=0x7f1660004470 comp rx=0 tx=0).stop 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 shutdown_connections 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f16540692a0 0x7f165406b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f166c0fe640 0x7f166c10dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 --2- 172.21.15.2:0/3069939959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166c0fef90 0x7f166c10e290 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 >> 172.21.15.2:0/3069939959 conn(0x7f166c0f6d20 msgr2=0x7f166c101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 shutdown_connections 2024-02-16T22:18:46.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:46.213+0000 7f1671481700 1 -- 172.21.15.2:0/3069939959 wait complete. 2024-02-16T22:18:46.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:45 smithi103 ceph-mon[36259]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:46.818 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-16T22:18:47.080 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:48.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:47 smithi002 ceph-mon[29154]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:48.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:47 smithi002 ceph-mon[29154]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:48.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:47 smithi103 ceph-mon[36259]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:48.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:47 smithi103 ceph-mon[36259]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:48.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.693+0000 7f9e5e90e700 1 -- 172.21.15.2:0/2378067531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fb870 msgr2=0x7f9e580fbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:48.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.693+0000 7f9e5e90e700 1 --2- 172.21.15.2:0/2378067531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fb870 0x7f9e580fbc80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9e48004940 tx=0x7f9e48009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:48.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.693+0000 7f9e5e90e700 1 -- 172.21.15.2:0/2378067531 shutdown_connections 2024-02-16T22:18:48.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.693+0000 7f9e5e90e700 1 --2- 172.21.15.2:0/2378067531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e580fc1c0 0x7f9e580fc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:48.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.693+0000 7f9e5e90e700 1 --2- 172.21.15.2:0/2378067531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fb870 0x7f9e580fbc80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.693+0000 7f9e5e90e700 1 -- 172.21.15.2:0/2378067531 >> 172.21.15.2:0/2378067531 conn(0x7f9e580f74e0 msgr2=0x7f9e580f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.693+0000 7f9e5e90e700 1 -- 172.21.15.2:0/2378067531 shutdown_connections 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.693+0000 7f9e5e90e700 1 -- 172.21.15.2:0/2378067531 wait complete. 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.693+0000 7f9e5e90e700 1 Processor -- start 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e5e90e700 1 -- start start 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e5e90e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e580fb870 0x7f9e5810fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e5e90e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fc1c0 0x7f9e58110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e5e90e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58110680 con 0x7f9e580fc1c0 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e5e90e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58110f20 con 0x7f9e580fb870 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fc1c0 0x7f9e58110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fc1c0 0x7f9e58110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43026/0 (socket says 172.21.15.2:43026) 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e577fe700 1 -- 172.21.15.2:0/3593953295 learned_addr learned my addr 172.21.15.2:0/3593953295 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e57fff700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e580fb870 0x7f9e5810fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:48.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e577fe700 1 -- 172.21.15.2:0/3593953295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e580fb870 msgr2=0x7f9e5810fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:48.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e577fe700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e580fb870 0x7f9e5810fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:48.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e577fe700 1 -- 172.21.15.2:0/3593953295 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e48005040 con 0x7f9e580fc1c0 2024-02-16T22:18:48.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e577fe700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fc1c0 0x7f9e58110080 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c830 tx=0x7f9e4c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:48.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e54ff9700 1 -- 172.21.15.2:0/3593953295 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e4c00a760 con 0x7f9e580fc1c0 2024-02-16T22:18:48.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e54ff9700 1 -- 172.21.15.2:0/3593953295 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e4c00ad90 con 0x7f9e580fc1c0 2024-02-16T22:18:48.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e58111200 con 0x7f9e580fc1c0 2024-02-16T22:18:48.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.694+0000 7f9e54ff9700 1 -- 172.21.15.2:0/3593953295 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e4c0113a0 con 0x7f9e580fc1c0 2024-02-16T22:18:48.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.695+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e58111740 con 0x7f9e580fc1c0 2024-02-16T22:18:48.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.695+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5804bed0 con 0x7f9e580fc1c0 2024-02-16T22:18:48.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.696+0000 7f9e54ff9700 1 -- 172.21.15.2:0/3593953295 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f9e4c012400 con 0x7f9e580fc1c0 2024-02-16T22:18:48.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.696+0000 7f9e54ff9700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9e400692a0 0x7f9e4006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:48.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.696+0000 7f9e57fff700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9e400692a0 0x7f9e4006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:48.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.696+0000 7f9e54ff9700 1 -- 172.21.15.2:0/3593953295 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9e4c086f10 con 0x7f9e580fc1c0 2024-02-16T22:18:48.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.697+0000 7f9e57fff700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9e400692a0 0x7f9e4006b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9e48004f90 tx=0x7f9e48009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:48.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.701+0000 7f9e54ff9700 1 -- 172.21.15.2:0/3593953295 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9e4c057010 con 0x7f9e580fc1c0 2024-02-16T22:18:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.897+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e58111ce0 con 0x7f9e400692a0 2024-02-16T22:18:48.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.899+0000 7f9e54ff9700 1 -- 172.21.15.2:0/3593953295 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f9e58111ce0 con 0x7f9e400692a0 2024-02-16T22:18:48.900 INFO:teuthology.orchestra.run.smithi002.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-16T22:18:48.901 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300EV480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-02-16T22:18:48.901 INFO:teuthology.orchestra.run.smithi002.stdout:smithi103 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204012Z400BGN 372G No 75s ago Has a FileSystem, LVM detected 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9e400692a0 msgr2=0x7f9e4006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9e400692a0 0x7f9e4006b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9e48004f90 tx=0x7f9e48009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fc1c0 msgr2=0x7f9e58110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fc1c0 0x7f9e58110080 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00c830 tx=0x7f9e4c004470 comp rx=0 tx=0).stop 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 shutdown_connections 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f9e400692a0 0x7f9e4006b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e580fb870 0x7f9e5810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 --2- 172.21.15.2:0/3593953295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e580fc1c0 0x7f9e58110080 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 >> 172.21.15.2:0/3593953295 conn(0x7f9e580f74e0 msgr2=0x7f9e58103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 shutdown_connections 2024-02-16T22:18:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:48.901+0000 7f9e5e90e700 1 -- 172.21.15.2:0/3593953295 wait complete. 2024-02-16T22:18:49.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:48 smithi002 ceph-mon[29154]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:49.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:48 smithi103 ceph-mon[36259]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:49.471 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-02-16T22:18:49.481 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-16T22:18:49.507 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:18:49.507 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-16T22:18:49.769 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:50.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:50 smithi002 ceph-mon[29154]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:51.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:50 smithi103 ceph-mon[36259]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.456+0000 7f78f0618700 1 -- 172.21.15.2:0/2347900826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fcef0 msgr2=0x7f78e80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.456+0000 7f78f0618700 1 --2- 172.21.15.2:0/2347900826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fcef0 0x7f78e80fd320 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f78e4004940 tx=0x7f78e4009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 -- 172.21.15.2:0/2347900826 shutdown_connections 2024-02-16T22:18:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 --2- 172.21.15.2:0/2347900826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78e80fe0e0 0x7f78e80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 --2- 172.21.15.2:0/2347900826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fcef0 0x7f78e80fd320 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 -- 172.21.15.2:0/2347900826 >> 172.21.15.2:0/2347900826 conn(0x7f78e80f8620 msgr2=0x7f78e80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 -- 172.21.15.2:0/2347900826 shutdown_connections 2024-02-16T22:18:51.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 -- 172.21.15.2:0/2347900826 wait complete. 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 Processor -- start 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 -- start start 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78e80fcef0 0x7f78e810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fe0e0 0x7f78e810deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e810e4b0 con 0x7f78e80fe0e0 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.457+0000 7f78f0618700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e810ed50 con 0x7f78e80fcef0 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78edbb3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fe0e0 0x7f78e810deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78edbb3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fe0e0 0x7f78e810deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39698/0 (socket says 172.21.15.2:39698) 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78edbb3700 1 -- 172.21.15.2:0/2849772854 learned_addr learned my addr 172.21.15.2:0/2849772854 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78ee3b4700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78e80fcef0 0x7f78e810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78edbb3700 1 -- 172.21.15.2:0/2849772854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78e80fcef0 msgr2=0x7f78e810d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78edbb3700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78e80fcef0 0x7f78e810d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78edbb3700 1 -- 172.21.15.2:0/2849772854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e4005040 con 0x7f78e80fe0e0 2024-02-16T22:18:51.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78edbb3700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fe0e0 0x7f78e810deb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f78d8002a50 tx=0x7f78d80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78deffd700 1 -- 172.21.15.2:0/2849772854 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78d8007cc0 con 0x7f78e80fe0e0 2024-02-16T22:18:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78deffd700 1 -- 172.21.15.2:0/2849772854 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78d800f040 con 0x7f78e80fe0e0 2024-02-16T22:18:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e810f030 con 0x7f78e80fe0e0 2024-02-16T22:18:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78deffd700 1 -- 172.21.15.2:0/2849772854 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78d800da30 con 0x7f78e80fe0e0 2024-02-16T22:18:51.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.458+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e810f570 con 0x7f78e80fe0e0 2024-02-16T22:18:51.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.459+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e804bed0 con 0x7f78e80fe0e0 2024-02-16T22:18:51.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.459+0000 7f78deffd700 1 -- 172.21.15.2:0/2849772854 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f78d800db90 con 0x7f78e80fe0e0 2024-02-16T22:18:51.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.460+0000 7f78deffd700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f78d40692a0 0x7f78d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:51.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.460+0000 7f78deffd700 1 -- 172.21.15.2:0/2849772854 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f78d8086b50 con 0x7f78e80fe0e0 2024-02-16T22:18:51.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.460+0000 7f78ee3b4700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f78d40692a0 0x7f78d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:51.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.460+0000 7f78ee3b4700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f78d40692a0 0x7f78d406b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f78e4004890 tx=0x7f78e4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:51.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.464+0000 7f78deffd700 1 -- 172.21.15.2:0/2849772854 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f78d8056c50 con 0x7f78e80fe0e0 2024-02-16T22:18:51.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:51.683+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f78e810fa10 con 0x7f78d40692a0 2024-02-16T22:18:51.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:51 smithi002 ceph-mon[29154]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:52.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:51 smithi103 ceph-mon[36259]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:53.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.016+0000 7f78deffd700 1 -- 172.21.15.2:0/2849772854 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f78e810fa10 con 0x7f78d40692a0 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f78d40692a0 msgr2=0x7f78d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f78d40692a0 0x7f78d406b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f78e4004890 tx=0x7f78e4009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fe0e0 msgr2=0x7f78e810deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fe0e0 0x7f78e810deb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f78d8002a50 tx=0x7f78d80046b0 comp rx=0 tx=0).stop 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 shutdown_connections 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f78d40692a0 0x7f78d406b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78e80fcef0 0x7f78e810d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 --2- 172.21.15.2:0/2849772854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e80fe0e0 0x7f78e810deb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 >> 172.21.15.2:0/2849772854 conn(0x7f78e80f8620 msgr2=0x7f78e81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 shutdown_connections 2024-02-16T22:18:53.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:53.020+0000 7f78f0618700 1 -- 172.21.15.2:0/2849772854 wait complete. 2024-02-16T22:18:53.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:52 smithi002 ceph-mon[29154]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:53.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:52 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-16T22:18:53.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:52 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-16T22:18:53.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:52 smithi002 ceph-mon[29154]: osdmap e41: 6 total, 6 up, 6 in 2024-02-16T22:18:53.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:52 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-16T22:18:53.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:52 smithi002 ceph-mon[29154]: pgmap v171: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:53.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:18:52.995+0000 7ff18798d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-16T22:18:53.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:52 smithi103 ceph-mon[36259]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:18:53.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:52 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-16T22:18:53.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:52 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-16T22:18:53.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:52 smithi103 ceph-mon[36259]: osdmap e41: 6 total, 6 up, 6 in 2024-02-16T22:18:53.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:52 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-16T22:18:53.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:52 smithi103 ceph-mon[36259]: pgmap v171: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:53.716 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: osdmap e42: 6 total, 6 up, 6 in 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: osdmap e43: 6 total, 6 up, 6 in 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: fsmap cephfs:0 2024-02-16T22:18:53.952 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: Saving service mds.cephfs spec with placement count:4 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi002.llkqse", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi002.llkqse", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: Deploying daemon mds.cephfs.smithi002.llkqse on smithi002 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:53.953 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:18:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: osdmap e42: 6 total, 6 up, 6 in 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: osdmap e43: 6 total, 6 up, 6 in 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: fsmap cephfs:0 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: Saving service mds.cephfs spec with placement count:4 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:54.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:18:54.327 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:54.327 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi002.llkqse", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-16T22:18:54.327 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi002.llkqse", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-16T22:18:54.327 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:54.327 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: Deploying daemon mds.cephfs.smithi002.llkqse on smithi002 2024-02-16T22:18:54.327 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:54.327 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:18:55.062 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:55.303 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:55 smithi002 ceph-mon[29154]: osdmap e44: 6 total, 6 up, 6 in 2024-02-16T22:18:55.375 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:55 smithi002 ceph-mon[29154]: pgmap v175: 65 pgs: 27 active+clean, 23 creating+peering, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:55 smithi103 ceph-mon[36259]: osdmap e44: 6 total, 6 up, 6 in 2024-02-16T22:18:55.376 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:55 smithi103 ceph-mon[36259]: pgmap v175: 65 pgs: 27 active+clean, 23 creating+peering, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:56.249 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:56 smithi002 ceph-mon[29154]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-16T22:18:56.249 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:56 smithi002 ceph-mon[29154]: osdmap e45: 6 total, 6 up, 6 in 2024-02-16T22:18:56.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:56 smithi103 ceph-mon[36259]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-16T22:18:56.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:56 smithi103 ceph-mon[36259]: osdmap e45: 6 total, 6 up, 6 in 2024-02-16T22:18:57.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.330+0000 7fef5ef9d700 1 -- 172.21.15.2:0/894071393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef58066c50 msgr2=0x7fef580ff930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:57.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.330+0000 7fef5ef9d700 1 --2- 172.21.15.2:0/894071393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef58066c50 0x7fef580ff930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fef48004940 tx=0x7fef48009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:57.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.330+0000 7fef5ef9d700 1 -- 172.21.15.2:0/894071393 shutdown_connections 2024-02-16T22:18:57.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.330+0000 7fef5ef9d700 1 --2- 172.21.15.2:0/894071393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef580fff00 0x7fef58102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.330+0000 7fef5ef9d700 1 --2- 172.21.15.2:0/894071393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef58066c50 0x7fef580ff930 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.330+0000 7fef5ef9d700 1 -- 172.21.15.2:0/894071393 >> 172.21.15.2:0/894071393 conn(0x7fef580f7300 msgr2=0x7fef580f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.330+0000 7fef5ef9d700 1 -- 172.21.15.2:0/894071393 shutdown_connections 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.330+0000 7fef5ef9d700 1 -- 172.21.15.2:0/894071393 wait complete. 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5ef9d700 1 Processor -- start 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5ef9d700 1 -- start start 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5ef9d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef58066c50 0x7fef5810d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5ef9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef580fff00 0x7fef5810dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5ef9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5810e270 con 0x7fef580fff00 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5ef9d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5810eb10 con 0x7fef58066c50 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5d79a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef580fff00 0x7fef5810dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5d79a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef580fff00 0x7fef5810dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39742/0 (socket says 172.21.15.2:39742) 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5d79a700 1 -- 172.21.15.2:0/943808848 learned_addr learned my addr 172.21.15.2:0/943808848 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5df9b700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef58066c50 0x7fef5810d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:57.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5d79a700 1 -- 172.21.15.2:0/943808848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef58066c50 msgr2=0x7fef5810d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:57.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5d79a700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef58066c50 0x7fef5810d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:57.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5d79a700 1 -- 172.21.15.2:0/943808848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef48005040 con 0x7fef580fff00 2024-02-16T22:18:57.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.331+0000 7fef5d79a700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef580fff00 0x7fef5810dc70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fef54002a50 tx=0x7fef54004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:57.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.332+0000 7fef4e7fc700 1 -- 172.21.15.2:0/943808848 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef54007cc0 con 0x7fef580fff00 2024-02-16T22:18:57.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.332+0000 7fef4e7fc700 1 -- 172.21.15.2:0/943808848 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef5400f040 con 0x7fef580fff00 2024-02-16T22:18:57.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.332+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5810edf0 con 0x7fef580fff00 2024-02-16T22:18:57.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.332+0000 7fef4e7fc700 1 -- 172.21.15.2:0/943808848 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef5400d9c0 con 0x7fef580fff00 2024-02-16T22:18:57.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.332+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5810f330 con 0x7fef580fff00 2024-02-16T22:18:57.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.333+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef581078a0 con 0x7fef580fff00 2024-02-16T22:18:57.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.333+0000 7fef4e7fc700 1 -- 172.21.15.2:0/943808848 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fef54009730 con 0x7fef580fff00 2024-02-16T22:18:57.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.333+0000 7fef4e7fc700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fef440692a0 0x7fef4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:18:57.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.333+0000 7fef4e7fc700 1 -- 172.21.15.2:0/943808848 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fef54086f30 con 0x7fef580fff00 2024-02-16T22:18:57.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.334+0000 7fef5df9b700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fef440692a0 0x7fef4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:18:57.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.334+0000 7fef5df9b700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fef440692a0 0x7fef4406b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fef4800b3b0 tx=0x7fef48009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:18:57.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.338+0000 7fef4e7fc700 1 -- 172.21.15.2:0/943808848 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fef54056c70 con 0x7fef580fff00 2024-02-16T22:18:57.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.580+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fef58105860 con 0x7fef580fff00 2024-02-16T22:18:57.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.580+0000 7fef4e7fc700 1 -- 172.21.15.2:0/943808848 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1245 (secure 0 0 0) 0x7fef54056800 con 0x7fef580fff00 2024-02-16T22:18:57.583 INFO:teuthology.orchestra.run.smithi002.stdout:e4 2024-02-16T22:18:57.583 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-16T22:18:57.583 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:18:57.583 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2024-02-16T22:18:57.583 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:18:57.583 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 4 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-02-16T22:18:52.996722+0000 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2024-02-16T22:18:57.009801+0000 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 60 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 0 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 1 2024-02-16T22:18:57.584 INFO:teuthology.orchestra.run.smithi002.stdout:in 0 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=14468} 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3] 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data disabled 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 0 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.llkqse{0:14468} state up:creating seq 1 addr [v2:172.21.15.2:6826/3499280462,v1:172.21.15.2:6827/3499280462] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:18:57.585 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fef440692a0 msgr2=0x7fef4406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fef440692a0 0x7fef4406b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fef4800b3b0 tx=0x7fef48009e30 comp rx=0 tx=0).stop 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef580fff00 msgr2=0x7fef5810dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef580fff00 0x7fef5810dc70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fef54002a50 tx=0x7fef54004600 comp rx=0 tx=0).stop 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 shutdown_connections 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fef440692a0 0x7fef4406b750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef58066c50 0x7fef5810d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 --2- 172.21.15.2:0/943808848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef580fff00 0x7fef5810dc70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 >> 172.21.15.2:0/943808848 conn(0x7fef580f7300 msgr2=0x7fef580f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 shutdown_connections 2024-02-16T22:18:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:18:57.583+0000 7fef5ef9d700 1 -- 172.21.15.2:0/943808848 wait complete. 2024-02-16T22:18:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 4 2024-02-16T22:18:57.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:57.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:57.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.kbizjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.kbizjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: Deploying daemon mds.cephfs.smithi103.kbizjo on smithi103 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: pgmap v177: 65 pgs: 39 active+clean, 23 creating+peering, 3 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.2:6826/3499280462,v1:172.21.15.2:6827/3499280462] up:boot 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: daemon mds.cephfs.smithi002.llkqse assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: Cluster is now healthy 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: fsmap cephfs:0 1 up:standby 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi002.llkqse"}]: dispatch 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:creating} 2024-02-16T22:18:57.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:57 smithi002 ceph-mon[29154]: daemon mds.cephfs.smithi002.llkqse is now active in filesystem cephfs as rank 0 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.kbizjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.kbizjo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: Deploying daemon mds.cephfs.smithi103.kbizjo on smithi103 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: pgmap v177: 65 pgs: 39 active+clean, 23 creating+peering, 3 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.2:6826/3499280462,v1:172.21.15.2:6827/3499280462] up:boot 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: daemon mds.cephfs.smithi002.llkqse assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: Cluster is now healthy 2024-02-16T22:18:57.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: fsmap cephfs:0 1 up:standby 2024-02-16T22:18:57.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi002.llkqse"}]: dispatch 2024-02-16T22:18:57.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:creating} 2024-02-16T22:18:57.827 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:57 smithi103 ceph-mon[36259]: daemon mds.cephfs.smithi002.llkqse is now active in filesystem cephfs as rank 0 2024-02-16T22:18:58.258 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-16T22:18:58.267 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:18:58.268 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-02-16T22:18:58.458 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:58 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/943808848' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:18:58.458 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:58 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.2:6826/3499280462,v1:172.21.15.2:6827/3499280462] up:active 2024-02-16T22:18:58.458 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:58 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 2024-02-16T22:18:58.531 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:18:58.653 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:58 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/943808848' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:18:58.653 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:58 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.2:6826/3499280462,v1:172.21.15.2:6827/3499280462] up:active 2024-02-16T22:18:58.653 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:58 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 2024-02-16T22:18:59.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:59 smithi002 ceph-mon[29154]: pgmap v178: 65 pgs: 45 active+clean, 20 creating+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 183 B/s wr, 0 op/s 2024-02-16T22:18:59.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:59 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:59.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:59 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:59.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:59 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:59.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:59 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi002.hhufhl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-16T22:18:59.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:59 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi002.hhufhl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-16T22:18:59.726 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:18:59 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:18:59.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:59 smithi103 ceph-mon[36259]: pgmap v178: 65 pgs: 45 active+clean, 20 creating+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 183 B/s wr, 0 op/s 2024-02-16T22:18:59.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:59 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:59.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:59 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:59.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:59 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:18:59.825 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:59 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi002.hhufhl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-16T22:18:59.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:59 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi002.hhufhl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-16T22:18:59.826 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:18:59 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:19:00.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.136+0000 7f412f21a700 1 -- 172.21.15.2:0/1914942364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 msgr2=0x7f41280fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:00.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.136+0000 7f412f21a700 1 --2- 172.21.15.2:0/1914942364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 0x7f41280fd420 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4114004940 tx=0x7f4114009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:00.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.136+0000 7f412f21a700 1 -- 172.21.15.2:0/1914942364 shutdown_connections 2024-02-16T22:19:00.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.136+0000 7f412f21a700 1 --2- 172.21.15.2:0/1914942364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 0x7f41280fd420 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:00.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.136+0000 7f412f21a700 1 --2- 172.21.15.2:0/1914942364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41280fbe40 0x7f41280fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:00.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.136+0000 7f412f21a700 1 -- 172.21.15.2:0/1914942364 >> 172.21.15.2:0/1914942364 conn(0x7f41280f7520 msgr2=0x7f41280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:00.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.137+0000 7f412f21a700 1 -- 172.21.15.2:0/1914942364 shutdown_connections 2024-02-16T22:19:00.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.137+0000 7f412f21a700 1 -- 172.21.15.2:0/1914942364 wait complete. 2024-02-16T22:19:00.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.137+0000 7f412f21a700 1 Processor -- start 2024-02-16T22:19:00.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.137+0000 7f412f21a700 1 -- start start 2024-02-16T22:19:00.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f412f21a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41280fbe40 0x7f4128110640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:00.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f412f21a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 0x7f4128112b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:00.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f412f21a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4128066d10 con 0x7f41280fcfb0 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f412f21a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4128066e50 con 0x7f41280fbe40 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f4127fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 0x7f4128112b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f4127fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 0x7f4128112b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39750/0 (socket says 172.21.15.2:39750) 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f4127fff700 1 -- 172.21.15.2:0/1290763869 learned_addr learned my addr 172.21.15.2:0/1290763869 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f412cfb6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41280fbe40 0x7f4128110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f4127fff700 1 -- 172.21.15.2:0/1290763869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41280fbe40 msgr2=0x7f4128110640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f4127fff700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41280fbe40 0x7f4128110640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f4127fff700 1 -- 172.21.15.2:0/1290763869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4114005040 con 0x7f41280fcfb0 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f4127fff700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 0x7f4128112b90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f41140093b0 tx=0x7f4114009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f41257fa700 1 -- 172.21.15.2:0/1290763869 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f411400e040 con 0x7f41280fcfb0 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4128113190 con 0x7f41280fcfb0 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f41257fa700 1 -- 172.21.15.2:0/1290763869 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f411400bcc0 con 0x7f41280fcfb0 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f41257fa700 1 -- 172.21.15.2:0/1290763869 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f411401c630 con 0x7f41280fcfb0 2024-02-16T22:19:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.138+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4128113680 con 0x7f41280fcfb0 2024-02-16T22:19:00.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.139+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412804bed0 con 0x7f41280fcfb0 2024-02-16T22:19:00.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.140+0000 7f41257fa700 1 -- 172.21.15.2:0/1290763869 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f4114013980 con 0x7f41280fcfb0 2024-02-16T22:19:00.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.140+0000 7f41257fa700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f41180692a0 0x7f411806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:00.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.140+0000 7f412cfb6700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f41180692a0 0x7f411806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:00.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.140+0000 7f41257fa700 1 -- 172.21.15.2:0/1290763869 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4114088520 con 0x7f41280fcfb0 2024-02-16T22:19:00.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.141+0000 7f412cfb6700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f41180692a0 0x7f411806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4128066990 tx=0x7f411c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:00.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.145+0000 7f41257fa700 1 -- 172.21.15.2:0/1290763869 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4114054850 con 0x7f41280fcfb0 2024-02-16T22:19:00.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.386+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f41281138a0 con 0x7f41280fcfb0 2024-02-16T22:19:00.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:00 smithi103 ceph-mon[36259]: Deploying daemon mds.cephfs.smithi002.hhufhl on smithi002 2024-02-16T22:19:00.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:00 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:boot 2024-02-16T22:19:00.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:00 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby 2024-02-16T22:19:00.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:00 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.kbizjo"}]: dispatch 2024-02-16T22:19:00.576 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:00 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby 2024-02-16T22:19:00.577 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:00 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1290763869' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-16T22:19:00.613 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:00 smithi002 ceph-mon[29154]: Deploying daemon mds.cephfs.smithi002.hhufhl on smithi002 2024-02-16T22:19:00.613 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:00 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:boot 2024-02-16T22:19:00.614 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:00 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby 2024-02-16T22:19:00.614 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:00 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.kbizjo"}]: dispatch 2024-02-16T22:19:00.614 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:00 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby 2024-02-16T22:19:00.614 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:00 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1290763869' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-16T22:19:00.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.888+0000 7f41257fa700 1 -- 172.21.15.2:0/1290763869 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7f41281138a0 con 0x7f41280fcfb0 2024-02-16T22:19:00.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f41180692a0 msgr2=0x7f411806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:00.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f41180692a0 0x7f411806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4128066990 tx=0x7f411c00b040 comp rx=0 tx=0).stop 2024-02-16T22:19:00.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 msgr2=0x7f4128112b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:00.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 0x7f4128112b90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f41140093b0 tx=0x7f4114009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:00.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 shutdown_connections 2024-02-16T22:19:00.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f41180692a0 0x7f411806b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:00.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41280fbe40 0x7f4128110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:00.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 --2- 172.21.15.2:0/1290763869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41280fcfb0 0x7f4128112b90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:00.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 >> 172.21.15.2:0/1290763869 conn(0x7f41280f7520 msgr2=0x7f4128100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:00.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 shutdown_connections 2024-02-16T22:19:00.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:00.890+0000 7f412f21a700 1 -- 172.21.15.2:0/1290763869 wait complete. 2024-02-16T22:19:01.521 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-16T22:19:01.531 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:19:01.532 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-02-16T22:19:01.992 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:02.170 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:01 smithi002 ceph-mon[29154]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-02-16T22:19:02.170 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:01 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1290763869' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-16T22:19:02.170 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:01 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby 2024-02-16T22:19:02.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:01 smithi103 ceph-mon[36259]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-02-16T22:19:02.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:01 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1290763869' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-16T22:19:02.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:01 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby 2024-02-16T22:19:03.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.744+0000 7f28d0f4c700 1 -- 172.21.15.2:0/688328679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fcd80 msgr2=0x7f28cc0fd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.744+0000 7f28d0f4c700 1 --2- 172.21.15.2:0/688328679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fcd80 0x7f28cc0fd1f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f28c0004940 tx=0x7f28c0009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.744+0000 7f28d0f4c700 1 -- 172.21.15.2:0/688328679 shutdown_connections 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.744+0000 7f28d0f4c700 1 --2- 172.21.15.2:0/688328679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fcd80 0x7f28cc0fd1f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.744+0000 7f28d0f4c700 1 --2- 172.21.15.2:0/688328679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28cc0fc430 0x7f28cc0fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.744+0000 7f28d0f4c700 1 -- 172.21.15.2:0/688328679 >> 172.21.15.2:0/688328679 conn(0x7f28cc0f6d20 msgr2=0x7f28cc0f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.744+0000 7f28d0f4c700 1 -- 172.21.15.2:0/688328679 shutdown_connections 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.744+0000 7f28d0f4c700 1 -- 172.21.15.2:0/688328679 wait complete. 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.745+0000 7f28d0f4c700 1 Processor -- start 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.745+0000 7f28d0f4c700 1 -- start start 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.745+0000 7f28d0f4c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fc430 0x7f28cc1109a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.745+0000 7f28d0f4c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28cc0fcd80 0x7f28cc112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.745+0000 7f28d0f4c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc0fbf20 con 0x7f28cc0fc430 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.745+0000 7f28d0f4c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc0fc060 con 0x7f28cc0fcd80 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.745+0000 7f28ca59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fc430 0x7f28cc1109a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.745+0000 7f28ca59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fc430 0x7f28cc1109a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47804/0 (socket says 172.21.15.2:47804) 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.745+0000 7f28ca59c700 1 -- 172.21.15.2:0/1489069662 learned_addr learned my addr 172.21.15.2:0/1489069662 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.746+0000 7f28ca59c700 1 -- 172.21.15.2:0/1489069662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28cc0fcd80 msgr2=0x7f28cc112ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.746+0000 7f28ca59c700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28cc0fcd80 0x7f28cc112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.746+0000 7f28ca59c700 1 -- 172.21.15.2:0/1489069662 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c0005040 con 0x7f28cc0fc430 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.746+0000 7f28ca59c700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fc430 0x7f28cc1109a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00b7c0 tx=0x7f28bc009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.746+0000 7f28baffd700 1 -- 172.21.15.2:0/1489069662 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28bc00f6f0 con 0x7f28cc0fc430 2024-02-16T22:19:03.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.746+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28cc1135a0 con 0x7f28cc0fc430 2024-02-16T22:19:03.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.746+0000 7f28baffd700 1 -- 172.21.15.2:0/1489069662 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28bc00fd20 con 0x7f28cc0fc430 2024-02-16T22:19:03.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.746+0000 7f28baffd700 1 -- 172.21.15.2:0/1489069662 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28bc0103a0 con 0x7f28cc0fc430 2024-02-16T22:19:03.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.746+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28cc113ae0 con 0x7f28cc0fc430 2024-02-16T22:19:03.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.747+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28cc106a00 con 0x7f28cc0fc430 2024-02-16T22:19:03.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.747+0000 7f28baffd700 1 -- 172.21.15.2:0/1489069662 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f28bc011400 con 0x7f28cc0fc430 2024-02-16T22:19:03.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.748+0000 7f28baffd700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f28b40692a0 0x7f28b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:03.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.748+0000 7f28baffd700 1 -- 172.21.15.2:0/1489069662 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f28bc086fc0 con 0x7f28cc0fc430 2024-02-16T22:19:03.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.748+0000 7f28c9d9b700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f28b40692a0 0x7f28b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:03.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.748+0000 7f28c9d9b700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f28b40692a0 0x7f28b406b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f28c0004f90 tx=0x7f28c0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:03.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.752+0000 7f28baffd700 1 -- 172.21.15.2:0/1489069662 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f28bc0532f0 con 0x7f28cc0fc430 2024-02-16T22:19:03.996 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:03 smithi002 ceph-mon[29154]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-02-16T22:19:03.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:03.994+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f28cc067690 con 0x7f28cc0fc430 2024-02-16T22:19:04.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:03 smithi103 ceph-mon[36259]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-02-16T22:19:04.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.887+0000 7f28baffd700 1 -- 172.21.15.2:0/1489069662 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7f28bc056910 con 0x7f28cc0fc430 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.890+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f28b40692a0 msgr2=0x7f28b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.890+0000 7f28d0f4c700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f28b40692a0 0x7f28b406b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f28c0004f90 tx=0x7f28c0009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.890+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fc430 msgr2=0x7f28cc1109a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.890+0000 7f28d0f4c700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fc430 0x7f28cc1109a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00b7c0 tx=0x7f28bc009220 comp rx=0 tx=0).stop 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.890+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 shutdown_connections 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.890+0000 7f28d0f4c700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f28b40692a0 0x7f28b406b750 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.890+0000 7f28d0f4c700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28cc0fc430 0x7f28cc1109a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.890+0000 7f28d0f4c700 1 --2- 172.21.15.2:0/1489069662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28cc0fcd80 0x7f28cc112ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.890+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 >> 172.21.15.2:0/1489069662 conn(0x7f28cc0f6d20 msgr2=0x7f28cc1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.891+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 shutdown_connections 2024-02-16T22:19:04.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:04.891+0000 7f28d0f4c700 1 -- 172.21.15.2:0/1489069662 wait complete. 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1489069662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.epsgnr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.epsgnr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1489069662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.2:6828/971745533,v1:172.21.15.2:6829/971745533] up:boot 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby-replay 1 up:standby 2024-02-16T22:19:05.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:04 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi002.hhufhl"}]: dispatch 2024-02-16T22:19:05.195 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1489069662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-16T22:19:05.195 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:05.195 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:05.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:05.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.epsgnr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-16T22:19:05.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi103.epsgnr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-16T22:19:05.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:19:05.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1489069662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-16T22:19:05.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.2:6828/971745533,v1:172.21.15.2:6829/971745533] up:boot 2024-02-16T22:19:05.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby-replay 1 up:standby 2024-02-16T22:19:05.196 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:04 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi002.hhufhl"}]: dispatch 2024-02-16T22:19:05.522 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-16T22:19:05.533 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:19:05.533 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-16T22:19:05.799 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:06.038 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:05 smithi103 ceph-mon[36259]: Deploying daemon mds.cephfs.smithi103.epsgnr on smithi103 2024-02-16T22:19:06.038 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:05 smithi103 ceph-mon[36259]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-16T22:19:06.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:05 smithi002 ceph-mon[29154]: Deploying daemon mds.cephfs.smithi103.epsgnr on smithi103 2024-02-16T22:19:06.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:05 smithi002 ceph-mon[29154]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-16T22:19:07.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.329+0000 7f3e19aec700 1 -- 172.21.15.2:0/822788144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fe0e0 msgr2=0x7f3e140fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.329+0000 7f3e19aec700 1 --2- 172.21.15.2:0/822788144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fe0e0 0x7f3e140fe550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3e08004940 tx=0x7f3e08009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.329+0000 7f3e19aec700 1 -- 172.21.15.2:0/822788144 shutdown_connections 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.329+0000 7f3e19aec700 1 --2- 172.21.15.2:0/822788144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fe0e0 0x7f3e140fe550 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.329+0000 7f3e19aec700 1 --2- 172.21.15.2:0/822788144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e140fcef0 0x7f3e140fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.329+0000 7f3e19aec700 1 -- 172.21.15.2:0/822788144 >> 172.21.15.2:0/822788144 conn(0x7f3e140f8620 msgr2=0x7f3e140faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.329+0000 7f3e19aec700 1 -- 172.21.15.2:0/822788144 shutdown_connections 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.329+0000 7f3e19aec700 1 -- 172.21.15.2:0/822788144 wait complete. 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.329+0000 7f3e19aec700 1 Processor -- start 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e19aec700 1 -- start start 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e19aec700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fcef0 0x7f3e1410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e19aec700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e140fe0e0 0x7f3e1410de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e19aec700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1410e430 con 0x7f3e140fcef0 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e19aec700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1410ecd0 con 0x7f3e140fe0e0 2024-02-16T22:19:07.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e137fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fcef0 0x7f3e1410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:07.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e137fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fcef0 0x7f3e1410d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47840/0 (socket says 172.21.15.2:47840) 2024-02-16T22:19:07.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e137fe700 1 -- 172.21.15.2:0/288112322 learned_addr learned my addr 172.21.15.2:0/288112322 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:07.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e12ffd700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e140fe0e0 0x7f3e1410de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e137fe700 1 -- 172.21.15.2:0/288112322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e140fe0e0 msgr2=0x7f3e1410de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e137fe700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e140fe0e0 0x7f3e1410de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e137fe700 1 -- 172.21.15.2:0/288112322 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e08005040 con 0x7f3e140fcef0 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.330+0000 7f3e137fe700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fcef0 0x7f3e1410d8f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3e04002a80 tx=0x7f3e040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.331+0000 7f3dfbfff700 1 -- 172.21.15.2:0/288112322 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e04007cc0 con 0x7f3e140fcef0 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.331+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1410efb0 con 0x7f3e140fcef0 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.331+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1410f4f0 con 0x7f3e140fcef0 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.331+0000 7f3dfbfff700 1 -- 172.21.15.2:0/288112322 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e0400f040 con 0x7f3e140fcef0 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.331+0000 7f3dfbfff700 1 -- 172.21.15.2:0/288112322 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e0400db30 con 0x7f3e140fcef0 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.332+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e000052e0 con 0x7f3e140fcef0 2024-02-16T22:19:07.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.332+0000 7f3dfbfff700 1 -- 172.21.15.2:0/288112322 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3e0400dc90 con 0x7f3e140fcef0 2024-02-16T22:19:07.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.333+0000 7f3dfbfff700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3dfc0692a0 0x7f3dfc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:07.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.333+0000 7f3dfbfff700 1 -- 172.21.15.2:0/288112322 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3e04086840 con 0x7f3e140fcef0 2024-02-16T22:19:07.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.333+0000 7f3e12ffd700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3dfc0692a0 0x7f3dfc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:07.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.333+0000 7f3e12ffd700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3dfc0692a0 0x7f3dfc06b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e08004f90 tx=0x7f3e08009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:07.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.341+0000 7f3dfbfff700 1 -- 172.21.15.2:0/288112322 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3e0404cbd0 con 0x7f3e140fcef0 2024-02-16T22:19:07.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.588+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3e00005590 con 0x7f3e140fcef0 2024-02-16T22:19:07.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.945+0000 7f3dfbfff700 1 -- 172.21.15.2:0/288112322 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v11) v1 ==== 168+0+0 (secure 0 0 0) 0x7f3e04056370 con 0x7f3e140fcef0 2024-02-16T22:19:07.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.947+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3dfc0692a0 msgr2=0x7f3dfc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:07.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.947+0000 7f3e19aec700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3dfc0692a0 0x7f3dfc06b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e08004f90 tx=0x7f3e08009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:07.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.947+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fcef0 msgr2=0x7f3e1410d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:07.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.947+0000 7f3e19aec700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fcef0 0x7f3e1410d8f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3e04002a80 tx=0x7f3e040046b0 comp rx=0 tx=0).stop 2024-02-16T22:19:07.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.948+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 shutdown_connections 2024-02-16T22:19:07.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.948+0000 7f3e19aec700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3dfc0692a0 0x7f3dfc06b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:07.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.948+0000 7f3e19aec700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e140fcef0 0x7f3e1410d8f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:07.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.948+0000 7f3e19aec700 1 --2- 172.21.15.2:0/288112322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e140fe0e0 0x7f3e1410de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:07.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.948+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 >> 172.21.15.2:0/288112322 conn(0x7f3e140f8620 msgr2=0x7f3e141013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:07.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.948+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 shutdown_connections 2024-02-16T22:19:07.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:07.948+0000 7f3e19aec700 1 -- 172.21.15.2:0/288112322 wait complete. 2024-02-16T22:19:07.951 INFO:teuthology.orchestra.run.smithi002.stderr:inline data enabled 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 88 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.103:6826/1310923282,v1:172.21.15.103:6827/1310923282] up:boot 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby-replay 2 up:standby 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.epsgnr"}]: dispatch 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:19:08.066 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:07 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/288112322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-16T22:19:08.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 88 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.103:6826/1310923282,v1:172.21.15.103:6827/1310923282] up:boot 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby-replay 2 up:standby 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi103.epsgnr"}]: dispatch 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:19:08.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:07 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/288112322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-16T22:19:08.662 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-16T22:19:08.672 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:19:08.672 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-16T22:19:09.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:standby-replay 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/288112322' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: Dropping low affinity active daemon mds.cephfs.smithi002.llkqse in favor of higher affinity standby. 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: Replacing daemon mds.cephfs.smithi002.llkqse as rank 0 with standby daemon mds.cephfs.smithi103.kbizjo 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby-replay 2 up:standby 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: osdmap e46: 6 total, 6 up, 6 in 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: fsmap cephfs:1/1 {0=cephfs.smithi103.kbizjo=up:replay} 2 up:standby 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:19:09.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:09.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:standby-replay 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/288112322' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: Dropping low affinity active daemon mds.cephfs.smithi002.llkqse in favor of higher affinity standby. 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: Replacing daemon mds.cephfs.smithi002.llkqse as rank 0 with standby daemon mds.cephfs.smithi103.kbizjo 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi002.llkqse=up:active} 1 up:standby-replay 2 up:standby 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: osdmap e46: 6 total, 6 up, 6 in 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: fsmap cephfs:1/1 {0=cephfs.smithi103.kbizjo=up:replay} 2 up:standby 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:19:09.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:08 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:09.597 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:09.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:09 smithi002 ceph-mon[29154]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 204 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-16T22:19:09.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:09 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.2:6826/3769723051,v1:172.21.15.2:6827/3769723051] up:boot 2024-02-16T22:19:09.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:09 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.2:6828/971745533,v1:172.21.15.2:6829/971745533] up:standby 2024-02-16T22:19:09.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:09 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:reconnect 2024-02-16T22:19:09.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:09 smithi002 ceph-mon[29154]: fsmap cephfs:1/1 {0=cephfs.smithi103.kbizjo=up:reconnect} 3 up:standby 2024-02-16T22:19:09.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi002.llkqse"}]: dispatch 2024-02-16T22:19:09.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:09.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:09 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:10.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:09 smithi103 ceph-mon[36259]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 204 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-16T22:19:10.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:09 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.2:6826/3769723051,v1:172.21.15.2:6827/3769723051] up:boot 2024-02-16T22:19:10.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:09 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.2:6828/971745533,v1:172.21.15.2:6829/971745533] up:standby 2024-02-16T22:19:10.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:09 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:reconnect 2024-02-16T22:19:10.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:09 smithi103 ceph-mon[36259]: fsmap cephfs:1/1 {0=cephfs.smithi103.kbizjo=up:reconnect} 3 up:standby 2024-02-16T22:19:10.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:09 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi002.llkqse"}]: dispatch 2024-02-16T22:19:10.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:09 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:10.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:09 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:10.958 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:10 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:rejoin 2024-02-16T22:19:10.958 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:10 smithi002 ceph-mon[29154]: fsmap cephfs:1/1 {0=cephfs.smithi103.kbizjo=up:rejoin} 3 up:standby 2024-02-16T22:19:11.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:10 smithi002 ceph-mon[29154]: daemon mds.cephfs.smithi103.kbizjo is now active in filesystem cephfs as rank 0 2024-02-16T22:19:11.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:10 smithi002 ceph-mon[29154]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-16T22:19:11.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:10 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:rejoin 2024-02-16T22:19:11.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:10 smithi103 ceph-mon[36259]: fsmap cephfs:1/1 {0=cephfs.smithi103.kbizjo=up:rejoin} 3 up:standby 2024-02-16T22:19:11.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:10 smithi103 ceph-mon[36259]: daemon mds.cephfs.smithi103.kbizjo is now active in filesystem cephfs as rank 0 2024-02-16T22:19:11.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:10 smithi103 ceph-mon[36259]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-16T22:19:12.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:11 smithi002 ceph-mon[29154]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-16T22:19:12.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:11 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:active 2024-02-16T22:19:12.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:11 smithi002 ceph-mon[29154]: mds.? [v2:172.21.15.103:6826/1310923282,v1:172.21.15.103:6827/1310923282] up:standby 2024-02-16T22:19:12.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:11 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi103.kbizjo=up:active} 3 up:standby 2024-02-16T22:19:12.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:11 smithi002 ceph-mon[29154]: fsmap cephfs:1 {0=cephfs.smithi103.kbizjo=up:active} 1 up:standby-replay 2 up:standby 2024-02-16T22:19:12.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:11 smithi103 ceph-mon[36259]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-16T22:19:12.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:11 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] up:active 2024-02-16T22:19:12.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:11 smithi103 ceph-mon[36259]: mds.? [v2:172.21.15.103:6826/1310923282,v1:172.21.15.103:6827/1310923282] up:standby 2024-02-16T22:19:12.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:11 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi103.kbizjo=up:active} 3 up:standby 2024-02-16T22:19:12.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:11 smithi103 ceph-mon[36259]: fsmap cephfs:1 {0=cephfs.smithi103.kbizjo=up:active} 1 up:standby-replay 2 up:standby 2024-02-16T22:19:12.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.455+0000 7f0a666ff700 1 -- 172.21.15.2:0/1346421398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 msgr2=0x7f0a600ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:12.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.455+0000 7f0a666ff700 1 --2- 172.21.15.2:0/1346421398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 0x7f0a600ffaf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0a50004940 tx=0x7f0a50009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:12.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.455+0000 7f0a666ff700 1 -- 172.21.15.2:0/1346421398 shutdown_connections 2024-02-16T22:19:12.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.455+0000 7f0a666ff700 1 --2- 172.21.15.2:0/1346421398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a601000c0 0x7f0a60102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:12.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 --2- 172.21.15.2:0/1346421398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 0x7f0a600ffaf0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:12.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 -- 172.21.15.2:0/1346421398 >> 172.21.15.2:0/1346421398 conn(0x7f0a600f7520 msgr2=0x7f0a600f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:12.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 -- 172.21.15.2:0/1346421398 shutdown_connections 2024-02-16T22:19:12.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 -- 172.21.15.2:0/1346421398 wait complete. 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 Processor -- start 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 -- start start 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 0x7f0a600fd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a601000c0 0x7f0a600fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a600fbff0 con 0x7f0a60066c70 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.456+0000 7f0a666ff700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a600fc160 con 0x7f0a601000c0 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 0x7f0a600fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 0x7f0a600fd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34768/0 (socket says 172.21.15.2:34768) 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5ffff700 1 -- 172.21.15.2:0/4250430848 learned_addr learned my addr 172.21.15.2:0/4250430848 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5f7fe700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a601000c0 0x7f0a600fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5ffff700 1 -- 172.21.15.2:0/4250430848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a601000c0 msgr2=0x7f0a600fb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5ffff700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a601000c0 0x7f0a600fb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5ffff700 1 -- 172.21.15.2:0/4250430848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a50005040 con 0x7f0a60066c70 2024-02-16T22:19:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5f7fe700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a601000c0 0x7f0a600fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-16T22:19:12.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5ffff700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 0x7f0a600fd570 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0a500093b0 tx=0x7f0a50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:12.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5cff9700 1 -- 172.21.15.2:0/4250430848 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a5000e040 con 0x7f0a60066c70 2024-02-16T22:19:12.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a600fc3e0 con 0x7f0a60066c70 2024-02-16T22:19:12.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5cff9700 1 -- 172.21.15.2:0/4250430848 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a5000bcc0 con 0x7f0a60066c70 2024-02-16T22:19:12.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a5cff9700 1 -- 172.21.15.2:0/4250430848 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a5001c630 con 0x7f0a60066c70 2024-02-16T22:19:12.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.457+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a601159b0 con 0x7f0a60066c70 2024-02-16T22:19:12.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.458+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a60107a70 con 0x7f0a60066c70 2024-02-16T22:19:12.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.459+0000 7f0a5cff9700 1 -- 172.21.15.2:0/4250430848 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f0a500139f0 con 0x7f0a60066c70 2024-02-16T22:19:12.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.459+0000 7f0a5cff9700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0a480692a0 0x7f0a4806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:12.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.459+0000 7f0a5f7fe700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0a480692a0 0x7f0a4806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:12.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.459+0000 7f0a5cff9700 1 -- 172.21.15.2:0/4250430848 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0a500885e0 con 0x7f0a60066c70 2024-02-16T22:19:12.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.459+0000 7f0a5f7fe700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0a480692a0 0x7f0a4806b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0a540041d0 tx=0x7f0a54009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.464+0000 7f0a5cff9700 1 -- 172.21.15.2:0/4250430848 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0a50058350 con 0x7f0a60066c70 2024-02-16T22:19:12.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.707+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0a60107490 con 0x7f0a60066c70 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.708+0000 7f0a5cff9700 1 -- 172.21.15.2:0/4250430848 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1830 (secure 0 0 0) 0x7f0a50057ee0 con 0x7f0a60066c70 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stdout:e16 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 16 2024-02-16T22:19:12.710 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-02-16T22:18:52.996722+0000 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2024-02-16T22:19:10.960547+0000 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 60 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 46 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 1 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:in 0 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24263} 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2024-02-16T22:19:12.711 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3] 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data enabled 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 1 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.kbizjo{0:24263} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.hhufhl{0:14486} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.2:6828/971745533,v1:172.21.15.2:6829/971745533] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.llkqse{-1:14496} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.2:6826/3769723051,v1:172.21.15.2:6827/3769723051] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:19:12.712 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.epsgnr{-1:24271} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.103:6826/1310923282,v1:172.21.15.103:6827/1310923282] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:19:12.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0a480692a0 msgr2=0x7f0a4806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:12.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0a480692a0 0x7f0a4806b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0a540041d0 tx=0x7f0a54009040 comp rx=0 tx=0).stop 2024-02-16T22:19:12.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 msgr2=0x7f0a600fd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:12.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 0x7f0a600fd570 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0a500093b0 tx=0x7f0a50009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:12.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 shutdown_connections 2024-02-16T22:19:12.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0a480692a0 0x7f0a4806b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:12.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a60066c70 0x7f0a600fd570 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:12.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 --2- 172.21.15.2:0/4250430848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a601000c0 0x7f0a600fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:12.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 >> 172.21.15.2:0/4250430848 conn(0x7f0a600f7520 msgr2=0x7f0a600f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:12.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 shutdown_connections 2024-02-16T22:19:12.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:12.711+0000 7f0a666ff700 1 -- 172.21.15.2:0/4250430848 wait complete. 2024-02-16T22:19:12.714 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 16 2024-02-16T22:19:13.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:12 smithi002 ceph-mon[29154]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-16T22:19:13.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:12 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/4250430848' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:19:13.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:12 smithi103 ceph-mon[36259]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-16T22:19:13.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:12 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/4250430848' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:19:13.355 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-16T22:19:13.710 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:15.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.311+0000 7f2de43ad700 1 -- 172.21.15.2:0/4160059455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fbe40 msgr2=0x7f2ddc0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:15.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.311+0000 7f2de43ad700 1 --2- 172.21.15.2:0/4160059455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fbe40 0x7f2ddc0fc250 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8004940 tx=0x7f2dd8009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:15.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.311+0000 7f2de43ad700 1 -- 172.21.15.2:0/4160059455 shutdown_connections 2024-02-16T22:19:15.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.311+0000 7f2de43ad700 1 --2- 172.21.15.2:0/4160059455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ddc0fcfb0 0x7f2ddc0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:15.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.311+0000 7f2de43ad700 1 --2- 172.21.15.2:0/4160059455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fbe40 0x7f2ddc0fc250 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.311+0000 7f2de43ad700 1 -- 172.21.15.2:0/4160059455 >> 172.21.15.2:0/4160059455 conn(0x7f2ddc0f74e0 msgr2=0x7f2ddc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.311+0000 7f2de43ad700 1 -- 172.21.15.2:0/4160059455 shutdown_connections 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.311+0000 7f2de43ad700 1 -- 172.21.15.2:0/4160059455 wait complete. 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.311+0000 7f2de43ad700 1 Processor -- start 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de43ad700 1 -- start start 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de43ad700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ddc0fbe40 0x7f2ddc112a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de43ad700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fcfb0 0x7f2ddc114fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de43ad700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc066350 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de43ad700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc066490 con 0x7f2ddc0fbe40 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de1948700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fcfb0 0x7f2ddc114fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de1948700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fcfb0 0x7f2ddc114fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34786/0 (socket says 172.21.15.2:34786) 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de1948700 1 -- 172.21.15.2:0/3677399504 learned_addr learned my addr 172.21.15.2:0/3677399504 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de2149700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ddc0fbe40 0x7f2ddc112a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de1948700 1 -- 172.21.15.2:0/3677399504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ddc0fbe40 msgr2=0x7f2ddc112a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de1948700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ddc0fbe40 0x7f2ddc112a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:15.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de1948700 1 -- 172.21.15.2:0/3677399504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd8005040 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2de1948700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fcfb0 0x7f2ddc114fb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0002a50 tx=0x7f2dd00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:15.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.312+0000 7f2dceffd700 1 -- 172.21.15.2:0/3677399504 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dd0007cc0 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.313+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc1154f0 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.313+0000 7f2dceffd700 1 -- 172.21.15.2:0/3677399504 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2dd000f040 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.313+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc1158f0 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.313+0000 7f2dceffd700 1 -- 172.21.15.2:0/3677399504 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dd000da30 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.314+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc00052e0 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.314+0000 7f2dceffd700 1 -- 172.21.15.2:0/3677399504 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2dd000db90 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.315+0000 7f2dceffd700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2dc80692a0 0x7f2dc806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:15.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.315+0000 7f2dceffd700 1 -- 172.21.15.2:0/3677399504 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2dd0086e80 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.315+0000 7f2de2149700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2dc80692a0 0x7f2dc806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:15.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.315+0000 7f2de2149700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2dc80692a0 0x7f2dc806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8004890 tx=0x7f2dd8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:15.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.321+0000 7f2dceffd700 1 -- 172.21.15.2:0/3677399504 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2dd0053160 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.563+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dc0005b10 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.563+0000 7f2dceffd700 1 -- 172.21.15.2:0/3677399504 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4690 (secure 0 0 0) 0x7f2dd0056780 con 0x7f2ddc0fcfb0 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2dc80692a0 msgr2=0x7f2dc806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2dc80692a0 0x7f2dc806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8004890 tx=0x7f2dd8009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fcfb0 msgr2=0x7f2ddc114fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fcfb0 0x7f2ddc114fb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0002a50 tx=0x7f2dd00046b0 comp rx=0 tx=0).stop 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 shutdown_connections 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2dc80692a0 0x7f2dc806b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ddc0fbe40 0x7f2ddc112a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 --2- 172.21.15.2:0/3677399504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ddc0fcfb0 0x7f2ddc114fb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 >> 172.21.15.2:0/3677399504 conn(0x7f2ddc0f74e0 msgr2=0x7f2ddc100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:15.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 shutdown_connections 2024-02-16T22:19:15.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:15.566+0000 7f2de43ad700 1 -- 172.21.15.2:0/3677399504 wait complete. 2024-02-16T22:19:15.569 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 16 2024-02-16T22:19:15.580 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-16T22:19:16.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:15 smithi002 ceph-mon[29154]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-16T22:19:16.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:16.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:16.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:19:16.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:19:16.173 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:16.174 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:15 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:19:16.174 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:15 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3677399504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-16T22:19:16.200 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-16T22:19:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:15 smithi103 ceph-mon[36259]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-16T22:19:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:19:16.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:19:16.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:16.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:15 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:19:16.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:15 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3677399504' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-16T22:19:17.125 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:18.166 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:17 smithi002 ceph-mon[29154]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-16T22:19:18.166 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:17 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:18.166 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:17 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:18.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:17 smithi103 ceph-mon[36259]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-16T22:19:18.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:17 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:18.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:17 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.499+0000 7f5c54ef3700 1 -- 172.21.15.2:0/489858470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fb5c0 msgr2=0x7f5c500fb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.499+0000 7f5c54ef3700 1 --2- 172.21.15.2:0/489858470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fb5c0 0x7f5c500fb9d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5c40004940 tx=0x7f5c40009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.499+0000 7f5c54ef3700 1 -- 172.21.15.2:0/489858470 shutdown_connections 2024-02-16T22:19:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.499+0000 7f5c54ef3700 1 --2- 172.21.15.2:0/489858470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c500fc870 0x7f5c500fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.499+0000 7f5c54ef3700 1 --2- 172.21.15.2:0/489858470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fb5c0 0x7f5c500fb9d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.499+0000 7f5c54ef3700 1 -- 172.21.15.2:0/489858470 >> 172.21.15.2:0/489858470 conn(0x7f5c500f6d20 msgr2=0x7f5c500f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.499+0000 7f5c54ef3700 1 -- 172.21.15.2:0/489858470 shutdown_connections 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c54ef3700 1 -- 172.21.15.2:0/489858470 wait complete. 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c54ef3700 1 Processor -- start 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c54ef3700 1 -- start start 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c54ef3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c500fb5c0 0x7f5c5010dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c54ef3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fc870 0x7f5c5010e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c54ef3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5010e860 con 0x7f5c500fc870 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c54ef3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5010eff0 con 0x7f5c500fb5c0 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c4dd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fc870 0x7f5c5010e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c4dd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fc870 0x7f5c5010e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34804/0 (socket says 172.21.15.2:34804) 2024-02-16T22:19:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.500+0000 7f5c4dd9b700 1 -- 172.21.15.2:0/1118160569 learned_addr learned my addr 172.21.15.2:0/1118160569 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c4e59c700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c500fb5c0 0x7f5c5010dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c4dd9b700 1 -- 172.21.15.2:0/1118160569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c500fb5c0 msgr2=0x7f5c5010dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c4dd9b700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c500fb5c0 0x7f5c5010dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c4dd9b700 1 -- 172.21.15.2:0/1118160569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c40005040 con 0x7f5c500fc870 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c4e59c700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c500fb5c0 0x7f5c5010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c4dd9b700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fc870 0x7f5c5010e260 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800c830 tx=0x7f5c38004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c46ffd700 1 -- 172.21.15.2:0/1118160569 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c3800a760 con 0x7f5c500fc870 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c46ffd700 1 -- 172.21.15.2:0/1118160569 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c3800ad90 con 0x7f5c500fc870 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5010f2d0 con 0x7f5c500fc870 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c46ffd700 1 -- 172.21.15.2:0/1118160569 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c380113a0 con 0x7f5c500fc870 2024-02-16T22:19:19.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.501+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5010f810 con 0x7f5c500fc870 2024-02-16T22:19:19.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.502+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c50107980 con 0x7f5c500fc870 2024-02-16T22:19:19.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.502+0000 7f5c46ffd700 1 -- 172.21.15.2:0/1118160569 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f5c38012400 con 0x7f5c500fc870 2024-02-16T22:19:19.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.503+0000 7f5c46ffd700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5c3c0692a0 0x7f5c3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:19.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.503+0000 7f5c46ffd700 1 -- 172.21.15.2:0/1118160569 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5c38087190 con 0x7f5c500fc870 2024-02-16T22:19:19.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.503+0000 7f5c4e59c700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5c3c0692a0 0x7f5c3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:19.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.503+0000 7f5c4e59c700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5c3c0692a0 0x7f5c3c06b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000b3b0 tx=0x7f5c40009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:19.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.507+0000 7f5c46ffd700 1 -- 172.21.15.2:0/1118160569 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5c38056fa0 con 0x7f5c500fc870 2024-02-16T22:19:19.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.807+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f5c5004bed0 con 0x7f5c500fc870 2024-02-16T22:19:19.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.807+0000 7f5c46ffd700 1 -- 172.21.15.2:0/1118160569 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v16) v1 ==== 78+0+99 (secure 0 0 0) 0x7f5c38056b30 con 0x7f5c500fc870 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5c3c0692a0 msgr2=0x7f5c3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5c3c0692a0 0x7f5c3c06b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000b3b0 tx=0x7f5c40009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fc870 msgr2=0x7f5c5010e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fc870 0x7f5c5010e260 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800c830 tx=0x7f5c38004470 comp rx=0 tx=0).stop 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 shutdown_connections 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5c3c0692a0 0x7f5c3c06b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c500fb5c0 0x7f5c5010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 --2- 172.21.15.2:0/1118160569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c500fc870 0x7f5c5010e260 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 >> 172.21.15.2:0/1118160569 conn(0x7f5c500f6d20 msgr2=0x7f5c50101e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 shutdown_connections 2024-02-16T22:19:19.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:19.810+0000 7f5c54ef3700 1 -- 172.21.15.2:0/1118160569 wait complete. 2024-02-16T22:19:19.822 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-16T22:19:20.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:19 smithi002 ceph-mon[29154]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-16T22:19:20.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:19 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1118160569' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-16T22:19:20.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:19 smithi103 ceph-mon[36259]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-16T22:19:20.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:19 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1118160569' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-16T22:19:20.829 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-16T22:19:20.841 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-16T22:19:20.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.951+0000 7f54ac95d640 1 -- 172.21.15.2:0/125448717 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a80ff440 msgr2=0x7f54a8101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:20.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.951+0000 7f54ac95d640 1 --2- 172.21.15.2:0/125448717 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a80ff440 0x7f54a8101820 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f54900048d0 tx=0x7f5490009da0 comp rx=0 tx=0).stop 2024-02-16T22:19:20.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.951+0000 7f54ac95d640 1 -- 172.21.15.2:0/125448717 shutdown_connections 2024-02-16T22:19:20.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.951+0000 7f54ac95d640 1 --2- 172.21.15.2:0/125448717 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a80ff440 0x7f54a8101820 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:20.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.951+0000 7f54ac95d640 1 --2- 172.21.15.2:0/125448717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54a8066350 0x7f54a80fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:20.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.951+0000 7f54ac95d640 1 -- 172.21.15.2:0/125448717 >> 172.21.15.2:0/125448717 conn(0x7f54a80f6910 msgr2=0x7f54a80f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:20.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.951+0000 7f54ac95d640 1 -- 172.21.15.2:0/125448717 shutdown_connections 2024-02-16T22:19:20.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.951+0000 7f54ac95d640 1 -- 172.21.15.2:0/125448717 wait complete. 2024-02-16T22:19:20.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54ac95d640 1 Processor -- start 2024-02-16T22:19:20.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54ac95d640 1 -- start start 2024-02-16T22:19:20.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54ac95d640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a8066350 0x7f54a810cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54ac95d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54a80ff440 0x7f54a810d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54ac95d640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a810d770 con 0x7f54a8066350 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54ac95d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8112180 con 0x7f54a80ff440 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54a659c640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a8066350 0x7f54a810cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54a659c640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a8066350 0x7f54a810cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34036/0 (socket says 172.21.15.2:34036) 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54a659c640 1 -- 172.21.15.2:0/3161382027 learned_addr learned my addr 172.21.15.2:0/3161382027 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54a659c640 1 -- 172.21.15.2:0/3161382027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54a80ff440 msgr2=0x7f54a810d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54a659c640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54a80ff440 0x7f54a810d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54a659c640 1 -- 172.21.15.2:0/3161382027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5490005040 con 0x7f54a8066350 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.952+0000 7f54a659c640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a8066350 0x7f54a810cbe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f549800c810 tx=0x7f5498004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.953+0000 7f549f7fe640 1 -- 172.21.15.2:0/3161382027 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f549800a850 con 0x7f54a8066350 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.953+0000 7f549f7fe640 1 -- 172.21.15.2:0/3161382027 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f549800ae80 con 0x7f54a8066350 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.953+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a8112320 con 0x7f54a8066350 2024-02-16T22:19:20.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.953+0000 7f549f7fe640 1 -- 172.21.15.2:0/3161382027 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54980115e0 con 0x7f54a8066350 2024-02-16T22:19:20.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.953+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a81127f0 con 0x7f54a8066350 2024-02-16T22:19:20.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.953+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a8106d80 con 0x7f54a8066350 2024-02-16T22:19:20.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.954+0000 7f549f7fe640 1 -- 172.21.15.2:0/3161382027 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f5498012400 con 0x7f54a8066350 2024-02-16T22:19:20.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.954+0000 7f549f7fe640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5494069210 0x7f549406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:20.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.955+0000 7f549dd9b640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5494069210 0x7f549406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:20.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.955+0000 7f549f7fe640 1 -- 172.21.15.2:0/3161382027 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5498087410 con 0x7f54a8066350 2024-02-16T22:19:20.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.955+0000 7f549dd9b640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5494069210 0x7f549406b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5490034040 tx=0x7f5490009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:20.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:20.959+0000 7f549f7fe640 1 -- 172.21.15.2:0/3161382027 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f54980536f0 con 0x7f54a8066350 2024-02-16T22:19:21.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.139+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54a8104dd0 con 0x7f54a8066350 2024-02-16T22:19:21.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.139+0000 7f549f7fe640 1 -- 172.21.15.2:0/3161382027 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14316 (secure 0 0 0) 0x7f5498056d10 con 0x7f54a8066350 2024-02-16T22:19:21.141 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:21.141 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":46,"fsid":"43c11c36-cd18-11ee-95ba-87774f69a715","created":"2024-02-16T22:11:59.286486+0000","modified":"2024-02-16T22:19:07.945873+0000","last_up_change":"2024-02-16T22:17:32.871791+0000","last_in_change":"2024-02-16T22:17:13.716253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-16T22:16:18.867363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-16T22:18:51.932008+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-16T22:18:52.233749+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"348575cf-dd0d-444c-bab9-bd51dcc9189c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6803","nonce":3509867548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6805","nonce":3509867548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6809","nonce":3509867548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3509867548},{"type":"v1","addr":"172.21.15.2:6807","nonce":3509867548}]},"public_addr":"172.21.15.2:6803/3509867548","cluster_addr":"172.21.15.2:6805/3509867548","heartbeat_back_addr":"172.21.15.2:6809/3509867548","heartbeat_front_addr":"172.21.15.2:6807/3509867548","state":["exists","up"]},{"osd":1,"uuid":"eaffda16-c0db-440f-9a1a-99e228b3dc19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6811","nonce":609822393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6813","nonce":609822393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6817","nonce":609822393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":609822393},{"type":"v1","addr":"172.21.15.2:6815","nonce":609822393}]},"public_addr":"172.21.15.2:6811/609822393","cluster_addr":"172.21.15.2:6813/609822393","heartbeat_back_addr":"172.21.15.2:6817/609822393","heartbeat_front_addr":"172.21.15.2:6815/609822393","state":["exists","up"]},{"osd":2,"uuid":"732c0a64-5183-42aa-8220-a12116e52442","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6819","nonce":2737615042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6821","nonce":2737615042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6825","nonce":2737615042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2737615042},{"type":"v1","addr":"172.21.15.2:6823","nonce":2737615042}]},"public_addr":"172.21.15.2:6819/2737615042","cluster_addr":"172.21.15.2:6821/2737615042","heartbeat_back_addr":"172.21.15.2:6825/2737615042","heartbeat_front_addr":"172.21.15.2:6823/2737615042","state":["exists","up"]},{"osd":3,"uuid":"09a182db-e441-45f8-9d13-61d69b7db54d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6801","nonce":425086385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6803","nonce":425086385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6807","nonce":425086385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":425086385},{"type":"v1","addr":"172.21.15.103:6805","nonce":425086385}]},"public_addr":"172.21.15.103:6801/425086385","cluster_addr":"172.21.15.103:6803/425086385","heartbeat_back_addr":"172.21.15.103:6807/425086385","heartbeat_front_addr":"172.21.15.103:6805/425086385","state":["exists","up"]},{"osd":4,"uuid":"ee9a7426-801d-420f-be92-cd18c21d11b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6809","nonce":420978374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6811","nonce":420978374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6815","nonce":420978374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":420978374},{"type":"v1","addr":"172.21.15.103:6813","nonce":420978374}]},"public_addr":"172.21.15.103:6809/420978374","cluster_addr":"172.21.15.103:6811/420978374","heartbeat_back_addr":"172.21.15.103:6815/420978374","heartbeat_front_addr":"172.21.15.103:6813/420978374","state":["exists","up"]},{"osd":5,"uuid":"68c72748-2d27-4ada-a35b-53e1bab20f9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6817","nonce":3241271941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6819","nonce":3241271941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6823","nonce":3241271941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":3241271941},{"type":"v1","addr":"172.21.15.103:6821","nonce":3241271941}]},"public_addr":"172.21.15.103:6817/3241271941","cluster_addr":"172.21.15.103:6819/3241271941","heartbeat_back_addr":"172.21.15.103:6823/3241271941","heartbeat_front_addr":"172.21.15.103:6821/3241271941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:15:21.168791+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:15:48.309630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:16:16.840477+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:16:40.949660+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:17:05.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-16T22:17:30.935853+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6800/395571706":"2024-02-17T22:12:27.545802+0000","172.21.15.2:6827/3499280462":"2024-02-17T22:19:07.945843+0000","172.21.15.2:6800/673699505":"2024-02-17T22:13:00.117629+0000","172.21.15.2:6801/395571706":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/3292480868":"2024-02-17T22:13:00.117629+0000","172.21.15.2:6826/3499280462":"2024-02-17T22:19:07.945843+0000","172.21.15.2:0/1318894983":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/495154782":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/3834071444":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/3921669523":"2024-02-17T22:12:27.545802+0000","172.21.15.2:0/135684194":"2024-02-17T22:13:00.117629+0000","172.21.15.2:6801/673699505":"2024-02-17T22:13:00.117629+0000","172.21.15.2:0/1364558863":"2024-02-17T22:14:08.529242+0000","172.21.15.2:0/504770625":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6800/2481260931":"2024-02-17T22:14:08.529242+0000","172.21.15.2:6801/2481260931":"2024-02-17T22:14:08.529242+0000","172.21.15.2:0/1058268540":"2024-02-17T22:14:08.529242+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5494069210 msgr2=0x7f549406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5494069210 0x7f549406b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5490034040 tx=0x7f5490009da0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a8066350 msgr2=0x7f54a810cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a8066350 0x7f54a810cbe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f549800c810 tx=0x7f5498004430 comp rx=0 tx=0).stop 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 shutdown_connections 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f5494069210 0x7f549406b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54a80ff440 0x7f54a810d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 --2- 172.21.15.2:0/3161382027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54a8066350 0x7f54a810cbe0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 >> 172.21.15.2:0/3161382027 conn(0x7f54a80f6910 msgr2=0x7f54a80f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 shutdown_connections 2024-02-16T22:19:21.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.142+0000 7f54ac95d640 1 -- 172.21.15.2:0/3161382027 wait complete. 2024-02-16T22:19:21.156 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-16T22:16:18.867363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-16T22:18:51.932008+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-16T22:18:52.233749+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-16T22:19:21.156 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-16T22:19:21.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.262+0000 7fa3f751d640 1 -- 172.21.15.2:0/765907982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00fce30 msgr2=0x7fa3f00ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.262+0000 7fa3f751d640 1 --2- 172.21.15.2:0/765907982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00fce30 0x7fa3f00ff210 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0048d0 tx=0x7fa3ec009da0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.262+0000 7fa3f751d640 1 -- 172.21.15.2:0/765907982 shutdown_connections 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.262+0000 7fa3f751d640 1 --2- 172.21.15.2:0/765907982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3f00ff750 0x7fa3f0101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.262+0000 7fa3f751d640 1 --2- 172.21.15.2:0/765907982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00fce30 0x7fa3f00ff210 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.262+0000 7fa3f751d640 1 -- 172.21.15.2:0/765907982 >> 172.21.15.2:0/765907982 conn(0x7fa3f00f68d0 msgr2=0x7fa3f00f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.262+0000 7fa3f751d640 1 -- 172.21.15.2:0/765907982 shutdown_connections 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.262+0000 7fa3f751d640 1 -- 172.21.15.2:0/765907982 wait complete. 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f751d640 1 Processor -- start 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f751d640 1 -- start start 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f751d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3f00fce30 0x7fa3f010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f751d640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00ff750 0x7fa3f010d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f751d640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f010d740 con 0x7fa3f00ff750 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f751d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0112190 con 0x7fa3f00fce30 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f4ab8640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00ff750 0x7fa3f010d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:21.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f4ab8640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00ff750 0x7fa3f010d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34056/0 (socket says 172.21.15.2:34056) 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f4ab8640 1 -- 172.21.15.2:0/3624927748 learned_addr learned my addr 172.21.15.2:0/3624927748 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f52b9640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3f00fce30 0x7fa3f010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f4ab8640 1 -- 172.21.15.2:0/3624927748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3f00fce30 msgr2=0x7fa3f010cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f4ab8640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3f00fce30 0x7fa3f010cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.263+0000 7fa3f4ab8640 1 -- 172.21.15.2:0/3624927748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec005040 con 0x7fa3f00ff750 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.264+0000 7fa3f4ab8640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00ff750 0x7fa3f010d0f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000c760 tx=0x7fa3e00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.264+0000 7fa3e5ffb640 1 -- 172.21.15.2:0/3624927748 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3e000a790 con 0x7fa3f00ff750 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.264+0000 7fa3e5ffb640 1 -- 172.21.15.2:0/3624927748 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3e000adc0 con 0x7fa3f00ff750 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.264+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f0112330 con 0x7fa3f00ff750 2024-02-16T22:19:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.264+0000 7fa3e5ffb640 1 -- 172.21.15.2:0/3624927748 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3e00115e0 con 0x7fa3f00ff750 2024-02-16T22:19:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.264+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f0112820 con 0x7fa3f00ff750 2024-02-16T22:19:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.265+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f0106e00 con 0x7fa3f00ff750 2024-02-16T22:19:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.265+0000 7fa3e5ffb640 1 -- 172.21.15.2:0/3624927748 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa3e0012400 con 0x7fa3f00ff750 2024-02-16T22:19:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.266+0000 7fa3e5ffb640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa3dc069210 0x7fa3dc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.266+0000 7fa3e5ffb640 1 -- 172.21.15.2:0/3624927748 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa3e00872f0 con 0x7fa3f00ff750 2024-02-16T22:19:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.266+0000 7fa3f52b9640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa3dc069210 0x7fa3dc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.266+0000 7fa3f52b9640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa3dc069210 0x7fa3dc06b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec000c00 tx=0x7fa3ec009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:21.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.270+0000 7fa3e5ffb640 1 -- 172.21.15.2:0/3624927748 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa3e0053670 con 0x7fa3f00ff750 2024-02-16T22:19:21.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.443+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3f002ab20 con 0x7fa3f00ff750 2024-02-16T22:19:21.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.444+0000 7fa3e5ffb640 1 -- 172.21.15.2:0/3624927748 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3e0056c90 con 0x7fa3f00ff750 2024-02-16T22:19:21.445 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-02-16T22:19:21.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa3dc069210 msgr2=0x7fa3dc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa3dc069210 0x7fa3dc06b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec000c00 tx=0x7fa3ec009da0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00ff750 msgr2=0x7fa3f010d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00ff750 0x7fa3f010d0f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000c760 tx=0x7fa3e00043c0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 shutdown_connections 2024-02-16T22:19:21.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa3dc069210 0x7fa3dc06b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3f00ff750 0x7fa3f010d0f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 --2- 172.21.15.2:0/3624927748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3f00fce30 0x7fa3f010cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 >> 172.21.15.2:0/3624927748 conn(0x7fa3f00f68d0 msgr2=0x7fa3f00f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:21.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 shutdown_connections 2024-02-16T22:19:21.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.446+0000 7fa3f751d640 1 -- 172.21.15.2:0/3624927748 wait complete. 2024-02-16T22:19:21.458 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-16T22:19:21.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.561+0000 7f2ce659d640 1 -- 172.21.15.2:0/2641225712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fc3f0 msgr2=0x7f2ce00fc860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.561+0000 7f2ce659d640 1 --2- 172.21.15.2:0/2641225712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fc3f0 0x7f2ce00fc860 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40048d0 tx=0x7f2cd4009da0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.561+0000 7f2ce659d640 1 -- 172.21.15.2:0/2641225712 shutdown_connections 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.561+0000 7f2ce659d640 1 --2- 172.21.15.2:0/2641225712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fc3f0 0x7f2ce00fc860 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.561+0000 7f2ce659d640 1 --2- 172.21.15.2:0/2641225712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce00fb1c0 0x7f2ce00fb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.561+0000 7f2ce659d640 1 -- 172.21.15.2:0/2641225712 >> 172.21.15.2:0/2641225712 conn(0x7f2ce00f6900 msgr2=0x7f2ce00f8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.561+0000 7f2ce659d640 1 -- 172.21.15.2:0/2641225712 shutdown_connections 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.561+0000 7f2ce659d640 1 -- 172.21.15.2:0/2641225712 wait complete. 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2ce659d640 1 Processor -- start 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2ce659d640 1 -- start start 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2ce659d640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fb1c0 0x7f2ce0065c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2ce659d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce00fc3f0 0x7f2ce0113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2ce659d640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce010d700 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2ce659d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce0114260 con 0x7f2ce00fc3f0 2024-02-16T22:19:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2cdffff640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fb1c0 0x7f2ce0065c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2cdffff640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fb1c0 0x7f2ce0065c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34066/0 (socket says 172.21.15.2:34066) 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2cdffff640 1 -- 172.21.15.2:0/1801202074 learned_addr learned my addr 172.21.15.2:0/1801202074 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2cdf7fe640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce00fc3f0 0x7f2ce0113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.562+0000 7f2cdffff640 1 -- 172.21.15.2:0/1801202074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce00fc3f0 msgr2=0x7f2ce0113d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.563+0000 7f2cdffff640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce00fc3f0 0x7f2ce0113d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.563+0000 7f2cdffff640 1 -- 172.21.15.2:0/1801202074 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd4005040 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.563+0000 7f2cdffff640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fb1c0 0x7f2ce0065c50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0002840 tx=0x7f2cd0004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.563+0000 7f2cdcff9640 1 -- 172.21.15.2:0/1801202074 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cd0007c30 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.563+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce01144c0 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.563+0000 7f2cdcff9640 1 -- 172.21.15.2:0/1801202074 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cd000f040 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.563+0000 7f2cdcff9640 1 -- 172.21.15.2:0/1801202074 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cd0009570 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.563+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce01149b0 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.564+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce0103d30 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.565+0000 7f2cdcff9640 1 -- 172.21.15.2:0/1801202074 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2cd000dc80 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.565+0000 7f2cdcff9640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2cc8069210 0x7f2cc806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.565+0000 7f2cdcff9640 1 -- 172.21.15.2:0/1801202074 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2cd0087060 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.565+0000 7f2cdf7fe640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2cc8069210 0x7f2cc806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.566+0000 7f2cdf7fe640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2cc8069210 0x7f2cc806b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0065a00 tx=0x7f2cd4009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:21.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.570+0000 7f2cdcff9640 1 -- 172.21.15.2:0/1801202074 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2cd0053340 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.746+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f2ce0114ea0 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.747+0000 7f2cdcff9640 1 -- 172.21.15.2:0/1801202074 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2cd0056960 con 0x7f2ce00fb1c0 2024-02-16T22:19:21.749 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-02-16T22:19:21.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2cc8069210 msgr2=0x7f2cc806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2cc8069210 0x7f2cc806b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0065a00 tx=0x7f2cd4009da0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fb1c0 msgr2=0x7f2ce0065c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fb1c0 0x7f2ce0065c50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0002840 tx=0x7f2cd0004570 comp rx=0 tx=0).stop 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 shutdown_connections 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2cc8069210 0x7f2cc806b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce00fc3f0 0x7f2ce0113d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 --2- 172.21.15.2:0/1801202074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ce00fb1c0 0x7f2ce0065c50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 >> 172.21.15.2:0/1801202074 conn(0x7f2ce00f6900 msgr2=0x7f2ce00ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 shutdown_connections 2024-02-16T22:19:21.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.749+0000 7f2ce659d640 1 -- 172.21.15.2:0/1801202074 wait complete. 2024-02-16T22:19:21.761 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-16T22:19:21.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1143151164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fb1a0 msgr2=0x7fcea40fb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 --2- 172.21.15.2:0/1143151164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fb1a0 0x7fcea40fb5b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fce94009c50 tx=0x7fce94004d10 comp rx=0 tx=0).stop 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1143151164 shutdown_connections 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 --2- 172.21.15.2:0/1143151164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea40fc3d0 0x7fcea40fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 --2- 172.21.15.2:0/1143151164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fb1a0 0x7fcea40fb5b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1143151164 >> 172.21.15.2:0/1143151164 conn(0x7fcea40f68a0 msgr2=0x7fcea40f8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1143151164 shutdown_connections 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1143151164 wait complete. 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 Processor -- start 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.865+0000 7fcea9c6e640 1 -- start start 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea9c6e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea40fb1a0 0x7fcea410cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea9c6e640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fc3d0 0x7fcea410d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea9c6e640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea410d720 con 0x7fcea40fc3d0 2024-02-16T22:19:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea9c6e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4112170 con 0x7fcea40fb1a0 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea2ffd640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fc3d0 0x7fcea410d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea2ffd640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fc3d0 0x7fcea410d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34074/0 (socket says 172.21.15.2:34074) 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea2ffd640 1 -- 172.21.15.2:0/1282642664 learned_addr learned my addr 172.21.15.2:0/1282642664 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea37fe640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea40fb1a0 0x7fcea410cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea2ffd640 1 -- 172.21.15.2:0/1282642664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea40fb1a0 msgr2=0x7fcea410cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea2ffd640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea40fb1a0 0x7fcea410cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea2ffd640 1 -- 172.21.15.2:0/1282642664 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce940098b0 con 0x7fcea40fc3d0 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.866+0000 7fcea2ffd640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fc3d0 0x7fcea410d0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fce98002a50 tx=0x7fce98004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.867+0000 7fce8bfff640 1 -- 172.21.15.2:0/1282642664 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce98007c30 con 0x7fcea40fc3d0 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.867+0000 7fce8bfff640 1 -- 172.21.15.2:0/1282642664 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce9800f040 con 0x7fcea40fc3d0 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.867+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea4112310 con 0x7fcea40fc3d0 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.867+0000 7fce8bfff640 1 -- 172.21.15.2:0/1282642664 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce9800db50 con 0x7fcea40fc3d0 2024-02-16T22:19:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.867+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea4112880 con 0x7fcea40fc3d0 2024-02-16T22:19:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.868+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea4103d10 con 0x7fcea40fc3d0 2024-02-16T22:19:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.868+0000 7fce8bfff640 1 -- 172.21.15.2:0/1282642664 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fce9801c400 con 0x7fcea40fc3d0 2024-02-16T22:19:21.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.869+0000 7fce8bfff640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fce8c069210 0x7fce8c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:21.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.869+0000 7fce8bfff640 1 -- 172.21.15.2:0/1282642664 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fce98086a20 con 0x7fcea40fc3d0 2024-02-16T22:19:21.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.869+0000 7fcea37fe640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fce8c069210 0x7fce8c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:21.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.869+0000 7fcea37fe640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fce8c069210 0x7fce8c06b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce94000f80 tx=0x7fce94037040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:21.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:21.873+0000 7fce8bfff640 1 -- 172.21.15.2:0/1282642664 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fce9804cbd0 con 0x7fcea40fc3d0 2024-02-16T22:19:22.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.047+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fcea4112aa0 con 0x7fcea40fc3d0 2024-02-16T22:19:22.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.047+0000 7fce8bfff640 1 -- 172.21.15.2:0/1282642664 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fce98052d30 con 0x7fcea40fc3d0 2024-02-16T22:19:22.049 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fce8c069210 msgr2=0x7fce8c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fce8c069210 0x7fce8c06b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce94000f80 tx=0x7fce94037040 comp rx=0 tx=0).stop 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fc3d0 msgr2=0x7fcea410d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fc3d0 0x7fcea410d0d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fce98002a50 tx=0x7fce98004600 comp rx=0 tx=0).stop 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 shutdown_connections 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fce8c069210 0x7fce8c06b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcea40fc3d0 0x7fcea410d0d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 --2- 172.21.15.2:0/1282642664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcea40fb1a0 0x7fcea410cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 >> 172.21.15.2:0/1282642664 conn(0x7fcea40f68a0 msgr2=0x7fcea4065630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 shutdown_connections 2024-02-16T22:19:22.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.049+0000 7fcea9c6e640 1 -- 172.21.15.2:0/1282642664 wait complete. 2024-02-16T22:19:22.060 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:21 smithi002 ceph-mon[29154]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-16T22:19:22.060 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:21 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3161382027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-16T22:19:22.060 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:21 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3624927748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-16T22:19:22.060 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:21 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1801202074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-16T22:19:22.061 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-16T22:19:22.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 -- 172.21.15.2:0/3034423608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 msgr2=0x7f19d40fc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:22.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 --2- 172.21.15.2:0/3034423608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 0x7f19d40fc8e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f19c80048d0 tx=0x7f19c8009da0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 -- 172.21.15.2:0/3034423608 shutdown_connections 2024-02-16T22:19:22.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 --2- 172.21.15.2:0/3034423608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 0x7f19d40fc8e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 --2- 172.21.15.2:0/3034423608 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19d40fb1a0 0x7f19d40fb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 -- 172.21.15.2:0/3034423608 >> 172.21.15.2:0/3034423608 conn(0x7f19d40f6960 msgr2=0x7f19d40f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:22.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 -- 172.21.15.2:0/3034423608 shutdown_connections 2024-02-16T22:19:22.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 -- 172.21.15.2:0/3034423608 wait complete. 2024-02-16T22:19:22.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 Processor -- start 2024-02-16T22:19:22.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.158+0000 7f19db86e640 1 -- start start 2024-02-16T22:19:22.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19db86e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19d40fb1a0 0x7f19d410cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:22.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19db86e640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 0x7f19d410d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:22.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19db86e640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d410d6c0 con 0x7f19d40fc470 2024-02-16T22:19:22.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19db86e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d4112110 con 0x7f19d40fb1a0 2024-02-16T22:19:22.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d8e09640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 0x7f19d410d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d8e09640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 0x7f19d410d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34094/0 (socket says 172.21.15.2:34094) 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d8e09640 1 -- 172.21.15.2:0/3671806098 learned_addr learned my addr 172.21.15.2:0/3671806098 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d960a640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19d40fb1a0 0x7f19d410cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d8e09640 1 -- 172.21.15.2:0/3671806098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19d40fb1a0 msgr2=0x7f19d410cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d8e09640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19d40fb1a0 0x7f19d410cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d8e09640 1 -- 172.21.15.2:0/3671806098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8005040 con 0x7f19d40fc470 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d8e09640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 0x7f19d410d070 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f19c802bda0 tx=0x7f19c8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d1ffb640 1 -- 172.21.15.2:0/3671806098 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c80094c0 con 0x7f19d40fc470 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d1ffb640 1 -- 172.21.15.2:0/3671806098 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19c8009620 con 0x7f19d40fc470 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d4112250 con 0x7f19d40fc470 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.159+0000 7f19d1ffb640 1 -- 172.21.15.2:0/3671806098 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19c803b640 con 0x7f19d40fc470 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.160+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d4112740 con 0x7f19d40fc470 2024-02-16T22:19:22.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.161+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d4104e10 con 0x7f19d40fc470 2024-02-16T22:19:22.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.161+0000 7f19d1ffb640 1 -- 172.21.15.2:0/3671806098 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f19c802f070 con 0x7f19d40fc470 2024-02-16T22:19:22.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.161+0000 7f19d1ffb640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f19bc069210 0x7f19bc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:22.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.162+0000 7f19d960a640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f19bc069210 0x7f19bc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:22.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.162+0000 7f19d1ffb640 1 -- 172.21.15.2:0/3671806098 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f19c80a9840 con 0x7f19d40fc470 2024-02-16T22:19:22.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.162+0000 7f19d960a640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f19bc069210 0x7f19bc06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f19cc004a70 tx=0x7f19cc00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:22.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.166+0000 7f19d1ffb640 1 -- 172.21.15.2:0/3671806098 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f19c8075aa0 con 0x7f19d40fc470 2024-02-16T22:19:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:21 smithi103 ceph-mon[36259]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-16T22:19:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:21 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3161382027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-16T22:19:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:21 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3624927748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-16T22:19:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:21 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1801202074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-16T22:19:22.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.378+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19d41128d0 con 0x7f19d40fc470 2024-02-16T22:19:22.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.379+0000 7f19d1ffb640 1 -- 172.21.15.2:0/3671806098 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4690 (secure 0 0 0) 0x7f19c80790c0 con 0x7f19d40fc470 2024-02-16T22:19:22.380 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:22.381 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14496,"name":"cephfs.smithi002.llkqse","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6827/3769723051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3769723051},{"type":"v1","addr":"172.21.15.2:6827","nonce":3769723051}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24271,"name":"cephfs.smithi103.epsgnr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.103:6827/1310923282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1310923282},{"type":"v1","addr":"172.21.15.103:6827","nonce":1310923282}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":16,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-16T22:18:52.996722+0000","modified":"2024-02-16T22:19:10.960547+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14486":{"gid":14486,"name":"cephfs.smithi002.hhufhl","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.2:6829/971745533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":971745533},{"type":"v1","addr":"172.21.15.2:6829","nonce":971745533}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24263":{"gid":24263,"name":"cephfs.smithi103.kbizjo","rank":0,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.103:6825/419104453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":419104453},{"type":"v1","addr":"172.21.15.103:6825","nonce":419104453}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-02-16T22:19:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 16 2024-02-16T22:19:22.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f19bc069210 msgr2=0x7f19bc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:22.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f19bc069210 0x7f19bc06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f19cc004a70 tx=0x7f19cc00b040 comp rx=0 tx=0).stop 2024-02-16T22:19:22.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 msgr2=0x7f19d410d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:22.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 0x7f19d410d070 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f19c802bda0 tx=0x7f19c8009da0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 shutdown_connections 2024-02-16T22:19:22.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f19bc069210 0x7f19bc06b6c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19d40fc470 0x7f19d410d070 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 --2- 172.21.15.2:0/3671806098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19d40fb1a0 0x7f19d410cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:22.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 >> 172.21.15.2:0/3671806098 conn(0x7f19d40f6960 msgr2=0x7f19d40ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:22.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 shutdown_connections 2024-02-16T22:19:22.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:22.381+0000 7f19db86e640 1 -- 172.21.15.2:0/3671806098 wait complete. 2024-02-16T22:19:22.398 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 16, 'max_mds': 1, 'flags': 50} 2024-02-16T22:19:22.398 INFO:teuthology.run_tasks:Running task kclient... 2024-02-16T22:19:22.421 INFO:tasks.kclient:Mounting kernel clients... 2024-02-16T22:19:22.421 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-16T22:19:22.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:22.421 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-02-16T22:19:22.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:22.437 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2024-02-16T22:19:22.508 INFO:teuthology.orchestra.run.smithi002.stderr:Cannot find device "ceph-brx" 2024-02-16T22:19:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-16T22:19:22.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:22.510 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-02-16T22:19:22.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:22.529 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-02-16T22:19:22.606 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-02-16T22:19:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-16T22:19:22.607 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-16T22:19:22.608 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-16T22:19:22.608 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-16T22:19:22.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph fs ls 2024-02-16T22:19:22.864 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:23.174 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:22 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1282642664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-16T22:19:23.174 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:22 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3671806098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-16T22:19:23.174 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:23.174 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:23.174 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:19:23.174 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:19:23.174 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:22 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:23.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:22 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1282642664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-16T22:19:23.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:22 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3671806098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-16T22:19:23.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:23.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:23.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:19:23.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:19:23.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:22 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:23 smithi002 ceph-mon[29154]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-16T22:19:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:19:24.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:23 smithi103 ceph-mon[36259]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-16T22:19:24.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:24.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:19:24.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.427+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4071203747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b8102f50 msgr2=0x7f39b8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:24.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.427+0000 7f39bd4c5700 1 --2- 172.21.15.2:0/4071203747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b8102f50 0x7f39b8103320 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f39a8004940 tx=0x7f39a8009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:24.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4071203747 shutdown_connections 2024-02-16T22:19:24.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 --2- 172.21.15.2:0/4071203747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b81038f0 0x7f39b80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:24.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 --2- 172.21.15.2:0/4071203747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b8102f50 0x7f39b8103320 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:24.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4071203747 >> 172.21.15.2:0/4071203747 conn(0x7f39b80f8ae0 msgr2=0x7f39b80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:24.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4071203747 shutdown_connections 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4071203747 wait complete. 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 Processor -- start 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 -- start start 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b8102f50 0x7f39b8113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b81038f0 0x7f39b8114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8114700 con 0x7f39b81038f0 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.428+0000 7f39bd4c5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b810d9a0 con 0x7f39b8102f50 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39b67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b81038f0 0x7f39b8114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39b67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b81038f0 0x7f39b8114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34108/0 (socket says 172.21.15.2:34108) 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39b67fc700 1 -- 172.21.15.2:0/4045593731 learned_addr learned my addr 172.21.15.2:0/4045593731 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39b6ffd700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b8102f50 0x7f39b8113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39b67fc700 1 -- 172.21.15.2:0/4045593731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b8102f50 msgr2=0x7f39b8113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39b67fc700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b8102f50 0x7f39b8113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:24.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39b67fc700 1 -- 172.21.15.2:0/4045593731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8005040 con 0x7f39b81038f0 2024-02-16T22:19:24.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39b67fc700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b81038f0 0x7f39b8114020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00b790 tx=0x7f39ac009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:24.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f399f7fe700 1 -- 172.21.15.2:0/4045593731 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39ac00f6f0 con 0x7f39b81038f0 2024-02-16T22:19:24.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b810dc80 con 0x7f39b81038f0 2024-02-16T22:19:24.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f399f7fe700 1 -- 172.21.15.2:0/4045593731 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39ac00fd20 con 0x7f39b81038f0 2024-02-16T22:19:24.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f399f7fe700 1 -- 172.21.15.2:0/4045593731 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39ac0103a0 con 0x7f39b81038f0 2024-02-16T22:19:24.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.429+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b810e1c0 con 0x7f39b81038f0 2024-02-16T22:19:24.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.430+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b80fc6e0 con 0x7f39b81038f0 2024-02-16T22:19:24.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.431+0000 7f399f7fe700 1 -- 172.21.15.2:0/4045593731 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f39ac011400 con 0x7f39b81038f0 2024-02-16T22:19:24.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.431+0000 7f399f7fe700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f39a00692a0 0x7f39a006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:24.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.431+0000 7f399f7fe700 1 -- 172.21.15.2:0/4045593731 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f39ac086eb0 con 0x7f39b81038f0 2024-02-16T22:19:24.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.431+0000 7f39b6ffd700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f39a00692a0 0x7f39a006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:24.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.432+0000 7f39b6ffd700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f39a00692a0 0x7f39a006b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f39a8004f90 tx=0x7f39a8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:24.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.435+0000 7f399f7fe700 1 -- 172.21.15.2:0/4045593731 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f39ac056cc0 con 0x7f39b81038f0 2024-02-16T22:19:24.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.671+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f39b804bed0 con 0x7f39b81038f0 2024-02-16T22:19:24.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.671+0000 7f399f7fe700 1 -- 172.21.15.2:0/4045593731 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7f39ac056850 con 0x7f39b81038f0 2024-02-16T22:19:24.673 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-16T22:19:24.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f39a00692a0 msgr2=0x7f39a006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:24.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f39a00692a0 0x7f39a006b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f39a8004f90 tx=0x7f39a8009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:24.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b81038f0 msgr2=0x7f39b8114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:24.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b81038f0 0x7f39b8114020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00b790 tx=0x7f39ac009220 comp rx=0 tx=0).stop 2024-02-16T22:19:24.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 shutdown_connections 2024-02-16T22:19:24.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f39a00692a0 0x7f39a006b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:24.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f39b8102f50 0x7f39b8113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:24.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 --2- 172.21.15.2:0/4045593731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b81038f0 0x7f39b8114020 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:24.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 >> 172.21.15.2:0/4045593731 conn(0x7f39b80f8ae0 msgr2=0x7f39b80fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:24.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 shutdown_connections 2024-02-16T22:19:24.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:24.673+0000 7f39bd4c5700 1 -- 172.21.15.2:0/4045593731 wait complete. 2024-02-16T22:19:24.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:24 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/4045593731' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-16T22:19:25.268 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-16T22:19:25.268 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-16T22:19:25.268 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-02-16T22:19:25.268 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-16T22:19:25.268 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-16T22:19:25.268 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-16T22:19:25.268 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-16T22:19:25.268 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-16T22:19:25.268 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-16T22:19:25.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:25.269 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-02-16T22:19:25.285 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-16T22:19:25.285 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-16T22:19:25.285 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-02-16T22:19:25.285 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-02-16T22:19:25.285 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 42282sec preferred_lft 42282sec 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-02-16T22:19:25.286 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-16T22:19:25.287 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-02-16T22:19:25.287 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-02-16T22:19:25.287 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-16T22:19:25.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:25.287 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-02-16T22:19:25.287 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-02-16T22:19:25.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2024-02-16T22:19:25.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2024-02-16T22:19:25.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2024-02-16T22:19:25.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-16T22:19:25.287 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-02-16T22:19:25.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:24 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/4045593731' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-16T22:19:25.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:25.473 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-16T22:19:25.500 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-02-16T22:19:25.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:25.502 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-02-16T22:19:25.578 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-02-16T22:19:25.578 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-16T22:19:25.578 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-02-16T22:19:25.578 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-02-16T22:19:25.578 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-16T22:19:25.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:25.579 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-02-16T22:19:25.579 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-02-16T22:19:25.579 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-16T22:19:25.579 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-16T22:19:25.579 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-16T22:19:25.579 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-02-16T22:19:25.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:25.877 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-02-16T22:19:25.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:25.934 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2024-02-16T22:19:25.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:25.949 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-02-16T22:19:25.949 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-02-16T22:19:25.949 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-16T22:19:25.949 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-16T22:19:25.949 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-02-16T22:19:26.035 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-16T22:19:26.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:26.035 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-02-16T22:19:26.035 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-02-16T22:19:26.035 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-16T22:19:26.035 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-16T22:19:26.035 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-16T22:19:26.036 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-16T22:19:26.036 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-16T22:19:26.036 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-02-16T22:19:26.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:26.133 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-02-16T22:19:26.133 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-02-16T22:19:26.134 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-02-16T22:19:26.134 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2024-02-16T22:19:26.134 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-02-16T22:19:26.158 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:25 smithi002 ceph-mon[29154]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-16T22:19:26.175 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-16T22:19:26.175 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-16T22:19:26.232 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-16T22:19:26.233 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-16T22:19:26.233 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-16T22:19:26.286 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-16T22:19:26.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:26.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-16T22:19:26.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:25 smithi103 ceph-mon[36259]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-02-16T22:19:26.381 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-16T22:19:26.381 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-16T22:19:26.381 INFO:teuthology.orchestra.run.smithi002.stdout:invalid new device string format 2024-02-16T22:19:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-16T22:19:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-16T22:19:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:invalid new device string format 2024-02-16T22:19:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: resolved to: "172.21.15.2:3300,172.21.15.103:3300" 2024-02-16T22:19:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-16T22:19:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-16T22:19:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-16T22:19:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:invalid new device string format 2024-02-16T22:19:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: resolved to: "172.21.15.2:3300,172.21.15.103:3300" 2024-02-16T22:19:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-16T22:19:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: resolved to: "172.21.15.2:6789,172.21.15.103:6789" 2024-02-16T22:19:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.2:6789,172.21.15.103:6789:/ 2024-02-16T22:19:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=43c11c36-cd18-11ee-95ba-87774f69a715" will pass to kernel 2024-02-16T22:19:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-16T22:19:26.497 INFO:teuthology.orchestra.run.smithi002.stdout: You just mounted a file system that supports labels which does not 2024-02-16T22:19:26.497 INFO:teuthology.orchestra.run.smithi002.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-16T22:19:26.497 INFO:teuthology.orchestra.run.smithi002.stdout: applications will generate AVC messages and not be allowed access to 2024-02-16T22:19:26.497 INFO:teuthology.orchestra.run.smithi002.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-16T22:19:26.499 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-16T22:19:26.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:26.499 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-16T22:19:26.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:26.527 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-16T22:19:26.590 INFO:teuthology.orchestra.run.smithi002.stdout:client24305 2024-02-16T22:19:26.591 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:19:26.591 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-16T22:19:26.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:26.647 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-16T22:19:26.710 INFO:teuthology.orchestra.run.smithi002.stdout:client24305 2024-02-16T22:19:26.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:26.710 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-16T22:19:26.773 INFO:teuthology.orchestra.run.smithi002.stdout:client24305 2024-02-16T22:19:26.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:26.773 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/43c11c36-cd18-11ee-95ba-87774f69a715.client24305/status') 2024-02-16T22:19:26.843 INFO:teuthology.orchestra.run.smithi002.stdout:instance: client.24305 (3)172.21.15.2:0/1620981693 2024-02-16T22:19:26.844 INFO:teuthology.orchestra.run.smithi002.stdout:blocklisted: false 2024-02-16T22:19:26.844 INFO:teuthology.orchestra.run.smithi002.stderr:0+1 records in 2024-02-16T22:19:26.844 INFO:teuthology.orchestra.run.smithi002.stderr:0+1 records out 2024-02-16T22:19:26.844 INFO:teuthology.orchestra.run.smithi002.stderr:70 bytes copied, 2.1732e-05 s, 3.2 MB/s 2024-02-16T22:19:26.845 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-16T22:19:26.845 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-16T22:19:26.845 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-16T22:19:26.845 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -- ceph fs ls 2024-02-16T22:19:27.154 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:27.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:26 smithi002 ceph-mon[29154]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-16T22:19:27.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:26 smithi103 ceph-mon[36259]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-16T22:19:28.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.662+0000 7f59fcfbf700 1 -- 172.21.15.2:0/2069567610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 msgr2=0x7f59f80fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.662+0000 7f59fcfbf700 1 --2- 172.21.15.2:0/2069567610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 0x7f59f80fbe70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f59e8004940 tx=0x7f59e8009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.662+0000 7f59fcfbf700 1 -- 172.21.15.2:0/2069567610 shutdown_connections 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.662+0000 7f59fcfbf700 1 --2- 172.21.15.2:0/2069567610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 0x7f59f80fbe70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.662+0000 7f59fcfbf700 1 --2- 172.21.15.2:0/2069567610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59f8101e60 0x7f59f8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.662+0000 7f59fcfbf700 1 -- 172.21.15.2:0/2069567610 >> 172.21.15.2:0/2069567610 conn(0x7f59f80f7a10 msgr2=0x7f59f80f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.662+0000 7f59fcfbf700 1 -- 172.21.15.2:0/2069567610 shutdown_connections 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.662+0000 7f59fcfbf700 1 -- 172.21.15.2:0/2069567610 wait complete. 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59fcfbf700 1 Processor -- start 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59fcfbf700 1 -- start start 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59fcfbf700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59f8101e60 0x7f59f8113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59fcfbf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 0x7f59f8113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59fcfbf700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f81146d0 con 0x7f59f8102800 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59fcfbf700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f810d970 con 0x7f59f8101e60 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59f5d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 0x7f59f8113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59f5d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 0x7f59f8113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34130/0 (socket says 172.21.15.2:34130) 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59f5d9b700 1 -- 172.21.15.2:0/3776702861 learned_addr learned my addr 172.21.15.2:0/3776702861 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59f5d9b700 1 -- 172.21.15.2:0/3776702861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59f8101e60 msgr2=0x7f59f8113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59f5d9b700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59f8101e60 0x7f59f8113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59f5d9b700 1 -- 172.21.15.2:0/3776702861 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e8005040 con 0x7f59f8102800 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.663+0000 7f59f5d9b700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 0x7f59f8113ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f59e8009150 tx=0x7f59e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.664+0000 7f59e6ffd700 1 -- 172.21.15.2:0/3776702861 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59e800e040 con 0x7f59f8102800 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.664+0000 7f59e6ffd700 1 -- 172.21.15.2:0/3776702861 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59e800bbf0 con 0x7f59f8102800 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.664+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f810dbf0 con 0x7f59f8102800 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.664+0000 7f59e6ffd700 1 -- 172.21.15.2:0/3776702861 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59e801c630 con 0x7f59f8102800 2024-02-16T22:19:28.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.664+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f810e130 con 0x7f59f8102800 2024-02-16T22:19:28.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.665+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f80fb610 con 0x7f59f8102800 2024-02-16T22:19:28.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.665+0000 7f59e6ffd700 1 -- 172.21.15.2:0/3776702861 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f59e800bd60 con 0x7f59f8102800 2024-02-16T22:19:28.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.665+0000 7f59e6ffd700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f59e00692a0 0x7f59e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:28.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.666+0000 7f59e6ffd700 1 -- 172.21.15.2:0/3776702861 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f59e80884f0 con 0x7f59f8102800 2024-02-16T22:19:28.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.666+0000 7f59f659c700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f59e00692a0 0x7f59e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:28.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.666+0000 7f59f659c700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f59e00692a0 0x7f59e006b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f59ec006fd0 tx=0x7f59ec006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:28.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.670+0000 7f59e6ffd700 1 -- 172.21.15.2:0/3776702861 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f59e8058300 con 0x7f59f8102800 2024-02-16T22:19:28.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.907+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f59f804bed0 con 0x7f59f8102800 2024-02-16T22:19:28.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.908+0000 7f59e6ffd700 1 -- 172.21.15.2:0/3776702861 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7f59e8057e90 con 0x7f59f8102800 2024-02-16T22:19:28.910 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-16T22:19:28.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f59e00692a0 msgr2=0x7f59e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f59e00692a0 0x7f59e006b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f59ec006fd0 tx=0x7f59ec006db0 comp rx=0 tx=0).stop 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 msgr2=0x7f59f8113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 0x7f59f8113ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f59e8009150 tx=0x7f59e8009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 shutdown_connections 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f59e00692a0 0x7f59e006b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59f8101e60 0x7f59f8113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 --2- 172.21.15.2:0/3776702861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59f8102800 0x7f59f8113ff0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 >> 172.21.15.2:0/3776702861 conn(0x7f59f80f7a10 msgr2=0x7f59f8065fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 shutdown_connections 2024-02-16T22:19:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:28.910+0000 7f59fcfbf700 1 -- 172.21.15.2:0/3776702861 wait complete. 2024-02-16T22:19:29.545 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-16T22:19:29.545 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-16T22:19:29.545 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-02-16T22:19:29.545 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-16T22:19:29.545 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-16T22:19:29.545 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-16T22:19:29.545 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-16T22:19:29.545 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-16T22:19:29.545 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-16T22:19:29.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:29.546 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-02-16T22:19:29.564 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-16T22:19:29.564 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-16T22:19:29.564 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-02-16T22:19:29.564 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-02-16T22:19:29.564 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42273sec preferred_lft 42273sec 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-02-16T22:19:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-02-16T22:19:29.566 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-16T22:19:29.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:29.566 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec bash -c ' 2024-02-16T22:19:29.566 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-02-16T22:19:29.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add name ceph-brx type bridge 2024-02-16T22:19:29.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr flush dev ceph-brx 2024-02-16T22:19:29.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set ceph-brx up 2024-02-16T22:19:29.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-16T22:19:29.566 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-02-16T22:19:29.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:29.749 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-16T22:19:29.780 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-02-16T22:19:29.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:29.781 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-02-16T22:19:29.855 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-02-16T22:19:29.855 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-16T22:19:29.855 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-16T22:19:29.855 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-16T22:19:29.855 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-16T22:19:29.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:29.856 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec bash -c ' 2024-02-16T22:19:29.856 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-02-16T22:19:29.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-16T22:19:29.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-16T22:19:29.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-16T22:19:29.856 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-02-16T22:19:29.921 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:29 smithi103 ceph-mon[36259]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:19:29.921 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:29 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3776702861' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-16T22:19:29.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:29 smithi002 ceph-mon[29154]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:19:29.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:29 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3776702861' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-16T22:19:30.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.105 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-02-16T22:19:30.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.161 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-02-16T22:19:30.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.216 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec bash -c ' 2024-02-16T22:19:30.216 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-02-16T22:19:30.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-16T22:19:30.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-16T22:19:30.216 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-02-16T22:19:30.301 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-16T22:19:30.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.302 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec bash -c ' 2024-02-16T22:19:30.302 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-02-16T22:19:30.302 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-16T22:19:30.302 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-16T22:19:30.302 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-16T22:19:30.302 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-16T22:19:30.302 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-16T22:19:30.302 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-02-16T22:19:30.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.409 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec bash -c ' 2024-02-16T22:19:30.409 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-02-16T22:19:30.409 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-02-16T22:19:30.409 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.0 master ceph-brx 2024-02-16T22:19:30.410 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-02-16T22:19:30.451 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-16T22:19:30.451 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-16T22:19:30.507 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-16T22:19:30.507 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-16T22:19:30.507 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-16T22:19:30.561 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-16T22:19:30.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-16T22:19:30.667 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-16T22:19:30.667 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-16T22:19:30.667 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-02-16T22:19:30.784 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-16T22:19:30.784 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-16T22:19:30.785 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-02-16T22:19:30.785 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.2:3300,172.21.15.103:3300" 2024-02-16T22:19:30.785 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-16T22:19:30.790 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-16T22:19:30.790 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-16T22:19:30.790 INFO:teuthology.orchestra.run.smithi103.stdout:invalid new device string format 2024-02-16T22:19:30.790 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.2:3300,172.21.15.103:3300" 2024-02-16T22:19:30.790 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-16T22:19:30.791 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.2:6789,172.21.15.103:6789" 2024-02-16T22:19:30.791 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.2:6789,172.21.15.103:6789:/ 2024-02-16T22:19:30.791 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=43c11c36-cd18-11ee-95ba-87774f69a715" will pass to kernel 2024-02-16T22:19:30.791 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-16T22:19:30.791 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-02-16T22:19:30.792 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-16T22:19:30.792 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-02-16T22:19:30.792 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-16T22:19:30.793 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-16T22:19:30.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.793 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-16T22:19:30.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.826 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-16T22:19:30.882 INFO:teuthology.orchestra.run.smithi103.stdout:client14538 2024-02-16T22:19:30.882 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:19:30.882 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-16T22:19:30.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.938 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-16T22:19:30.994 INFO:teuthology.orchestra.run.smithi103.stdout:client14538 2024-02-16T22:19:30.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:30.995 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-16T22:19:31.051 INFO:teuthology.orchestra.run.smithi103.stdout:client14538 2024-02-16T22:19:31.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:19:31.051 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/43c11c36-cd18-11ee-95ba-87774f69a715.client14538/status') 2024-02-16T22:19:31.118 INFO:teuthology.orchestra.run.smithi103.stdout:instance: client.14538 (3)172.21.15.103:0/635178128 2024-02-16T22:19:31.118 INFO:teuthology.orchestra.run.smithi103.stdout:blocklisted: false 2024-02-16T22:19:31.118 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records in 2024-02-16T22:19:31.118 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records out 2024-02-16T22:19:31.118 INFO:teuthology.orchestra.run.smithi103.stderr:71 bytes copied, 2.133e-05 s, 3.3 MB/s 2024-02-16T22:19:31.119 INFO:teuthology.run_tasks:Running task print... 2024-02-16T22:19:31.129 INFO:teuthology.task.print:**** done client 2024-02-16T22:19:31.129 INFO:teuthology.run_tasks:Running task parallel... 2024-02-16T22:19:31.180 INFO:teuthology.task.parallel:starting parallel... 2024-02-16T22:19:31.180 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-16T22:19:31.181 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-16T22:19:31.181 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:19:31.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-02-16T22:19:31.182 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-16T22:19:31.182 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-16T22:19:31.185 INFO:tasks.workunit:Pulling workunits from ref ed7ab725e1a43615009dfb74a42bf33d80081bc8 2024-02-16T22:19:31.185 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-16T22:19:31.185 INFO:tasks.workunit:timeout=3h 2024-02-16T22:19:31.185 INFO:tasks.workunit:cleanup=True 2024-02-16T22:19:31.185 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-16T22:19:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-16T22:19:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-16T22:19:31.206 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 53h/83d Inode: 1 Links: 2 2024-02-16T22:19:31.206 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:19:31.206 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-16T22:19:31.206 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-16T22:19:31.206 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-02-16 22:18:57.014065491 +0000 2024-02-16T22:19:31.206 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-02-16 22:19:26.524207113 +0000 2024-02-16T22:19:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-02-16 22:18:57.014065491 +0000 2024-02-16T22:19:31.207 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-16T22:19:31.207 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-16T22:19:31.278 DEBUG:teuthology.orchestra.run.smithi103:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-16T22:19:31.294 INFO:teuthology.orchestra.run.smithi103.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-16T22:19:31.294 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-02-16T22:19:31.294 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-02-16T22:19:31.294 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-16T22:19:31.294 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-16T22:19:31.294 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-16T22:19:31.294 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-02-16 22:19:31.272106605 +0000 2024-02-16T22:19:31.294 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-02-16 22:19:31.272106605 +0000 2024-02-16T22:19:31.294 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-02-16 22:18:57.014065491 +0000 2024-02-16T22:19:31.294 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-16T22:19:31.295 DEBUG:teuthology.orchestra.run.smithi103:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-16T22:19:31.364 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ed7ab725e1a43615009dfb74a42bf33d80081bc8 2024-02-16T22:19:31.365 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout ed7ab725e1a43615009dfb74a42bf33d80081bc8 2024-02-16T22:19:31.380 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-16T22:19:31.420 INFO:tasks.workunit.client.1.smithi103.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-16T22:19:31.459 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:32.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:31 smithi002 ceph-mon[29154]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-02-16T22:19:32.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:31 smithi103 ceph-mon[36259]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-02-16T22:19:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.041+0000 7ff9a7f05700 1 -- 172.21.15.2:0/790773970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9a01003a0 msgr2=0x7ff9a0102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.041+0000 7ff9a7f05700 1 --2- 172.21.15.2:0/790773970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9a01003a0 0x7ff9a0102780 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff990004940 tx=0x7ff990009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.041+0000 7ff9a7f05700 1 -- 172.21.15.2:0/790773970 shutdown_connections 2024-02-16T22:19:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.041+0000 7ff9a7f05700 1 --2- 172.21.15.2:0/790773970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9a01003a0 0x7ff9a0102780 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.041+0000 7ff9a7f05700 1 --2- 172.21.15.2:0/790773970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a00fda80 0x7ff9a00ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.041+0000 7ff9a7f05700 1 -- 172.21.15.2:0/790773970 >> 172.21.15.2:0/790773970 conn(0x7ff9a00f74e0 msgr2=0x7ff9a00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.041+0000 7ff9a7f05700 1 -- 172.21.15.2:0/790773970 shutdown_connections 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.041+0000 7ff9a7f05700 1 -- 172.21.15.2:0/790773970 wait complete. 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a7f05700 1 Processor -- start 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a7f05700 1 -- start start 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a7f05700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a00fda80 0x7ff9a010d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a7f05700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9a01003a0 0x7ff9a010de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a7f05700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a010e490 con 0x7ff9a01003a0 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a7f05700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a010ed30 con 0x7ff9a00fda80 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a54a0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9a01003a0 0x7ff9a010de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a54a0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9a01003a0 0x7ff9a010de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55298/0 (socket says 172.21.15.2:55298) 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a54a0700 1 -- 172.21.15.2:0/1732634979 learned_addr learned my addr 172.21.15.2:0/1732634979 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a54a0700 1 -- 172.21.15.2:0/1732634979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a00fda80 msgr2=0x7ff9a010d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a54a0700 1 --2- 172.21.15.2:0/1732634979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9a00fda80 0x7ff9a010d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a54a0700 1 -- 172.21.15.2:0/1732634979 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff990005040 con 0x7ff9a01003a0 2024-02-16T22:19:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.042+0000 7ff9a54a0700 1 --2- 172.21.15.2:0/1732634979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9a01003a0 0x7ff9a010de90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff9900093b0 tx=0x7ff990009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.043+0000 7ff9967fc700 1 -- 172.21.15.2:0/1732634979 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff99000e040 con 0x7ff9a01003a0 2024-02-16T22:19:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.043+0000 7ff9a7f05700 1 -- 172.21.15.2:0/1732634979 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a010efb0 con 0x7ff9a01003a0 2024-02-16T22:19:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.043+0000 7ff9967fc700 1 -- 172.21.15.2:0/1732634979 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff99000bcc0 con 0x7ff9a01003a0 2024-02-16T22:19:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.043+0000 7ff9967fc700 1 -- 172.21.15.2:0/1732634979 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff99001c630 con 0x7ff9a01003a0 2024-02-16T22:19:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.043+0000 7ff9a7f05700 1 -- 172.21.15.2:0/1732634979 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a010f4f0 con 0x7ff9a01003a0 2024-02-16T22:19:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.044+0000 7ff9a7f05700 1 -- 172.21.15.2:0/1732634979 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a0107a70 con 0x7ff9a01003a0 2024-02-16T22:19:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.044+0000 7ff9967fc700 1 -- 172.21.15.2:0/1732634979 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff9900139f0 con 0x7ff9a01003a0 2024-02-16T22:19:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.044+0000 7ff9967fc700 1 --2- 172.21.15.2:0/1732634979 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff98c0692a0 0x7ff98c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.045+0000 7ff9967fc700 1 -- 172.21.15.2:0/1732634979 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff990088510 con 0x7ff9a01003a0 2024-02-16T22:19:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.045+0000 7ff9a5ca1700 1 --2- 172.21.15.2:0/1732634979 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff98c0692a0 0x7ff98c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.045+0000 7ff9a5ca1700 1 --2- 172.21.15.2:0/1732634979 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ff98c0692a0 0x7ff98c06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff99c004d00 tx=0x7ff99c009ce0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.049+0000 7ff9967fc700 1 -- 172.21.15.2:0/1732634979 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff990058320 con 0x7ff9a01003a0 2024-02-16T22:19:33.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.242+0000 7ff9a7f05700 1 -- 172.21.15.2:0/1732634979 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7ff9a0105ae0 con 0x7ff9a01003a0 2024-02-16T22:19:33.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:33.243+0000 7ff9967fc700 1 -- 172.21.15.2:0/1732634979 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7ff990057eb0 con 0x7ff9a01003a0 2024-02-16T22:19:33.246 INFO:teuthology.orchestra.run.smithi002.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2024-02-16T22:19:33.837 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-16T22:19:33.838 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:19:33.838 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-16T22:19:33.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:33 smithi002 ceph-mon[29154]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-02-16T22:19:34.143 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:34.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:33 smithi103 ceph-mon[36259]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-02-16T22:19:35.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.810+0000 7ffb9b114700 1 -- 172.21.15.2:0/3081348142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 msgr2=0x7ffb940fe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.810+0000 7ffb9b114700 1 --2- 172.21.15.2:0/3081348142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 0x7ffb940fe680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ffb88004940 tx=0x7ffb88009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.810+0000 7ffb9b114700 1 -- 172.21.15.2:0/3081348142 shutdown_connections 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.810+0000 7ffb9b114700 1 --2- 172.21.15.2:0/3081348142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 0x7ffb940fe680 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.810+0000 7ffb9b114700 1 --2- 172.21.15.2:0/3081348142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb940fb1a0 0x7ffb940fb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.810+0000 7ffb9b114700 1 -- 172.21.15.2:0/3081348142 >> 172.21.15.2:0/3081348142 conn(0x7ffb940f6c00 msgr2=0x7ffb940f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.810+0000 7ffb9b114700 1 -- 172.21.15.2:0/3081348142 shutdown_connections 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.810+0000 7ffb9b114700 1 -- 172.21.15.2:0/3081348142 wait complete. 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.810+0000 7ffb9b114700 1 Processor -- start 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb9b114700 1 -- start start 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb9b114700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb940fb1a0 0x7ffb9410d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb9b114700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 0x7ffb9410ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb9b114700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9410e3f0 con 0x7ffb940fe230 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb9b114700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9410ec90 con 0x7ffb940fb1a0 2024-02-16T22:19:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb93fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 0x7ffb9410ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb98eb0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb940fb1a0 0x7ffb9410d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb93fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 0x7ffb9410ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55314/0 (socket says 172.21.15.2:55314) 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb93fff700 1 -- 172.21.15.2:0/1258714088 learned_addr learned my addr 172.21.15.2:0/1258714088 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb93fff700 1 -- 172.21.15.2:0/1258714088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb940fb1a0 msgr2=0x7ffb9410d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb93fff700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb940fb1a0 0x7ffb9410d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.811+0000 7ffb93fff700 1 -- 172.21.15.2:0/1258714088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb88005040 con 0x7ffb940fe230 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.812+0000 7ffb93fff700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 0x7ffb9410ddf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ffb880093b0 tx=0x7ffb88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.812+0000 7ffb917fa700 1 -- 172.21.15.2:0/1258714088 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb8800e040 con 0x7ffb940fe230 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.812+0000 7ffb917fa700 1 -- 172.21.15.2:0/1258714088 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb8800bcc0 con 0x7ffb940fe230 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.812+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9410ef10 con 0x7ffb940fe230 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.812+0000 7ffb917fa700 1 -- 172.21.15.2:0/1258714088 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb8801c630 con 0x7ffb940fe230 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.812+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9410f450 con 0x7ffb940fe230 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.813+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9404bed0 con 0x7ffb940fe230 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.813+0000 7ffb917fa700 1 -- 172.21.15.2:0/1258714088 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ffb88013980 con 0x7ffb940fe230 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.814+0000 7ffb917fa700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffb7c0692a0 0x7ffb7c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:35.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.814+0000 7ffb917fa700 1 -- 172.21.15.2:0/1258714088 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ffb88088550 con 0x7ffb940fe230 2024-02-16T22:19:35.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.814+0000 7ffb98eb0700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffb7c0692a0 0x7ffb7c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:35.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.814+0000 7ffb98eb0700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffb7c0692a0 0x7ffb7c06b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb94065cb0 tx=0x7ffb8400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:35.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:35.818+0000 7ffb917fa700 1 -- 172.21.15.2:0/1258714088 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ffb880582c0 con 0x7ffb940fe230 2024-02-16T22:19:35.899 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:35 smithi002 ceph-mon[29154]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-02-16T22:19:36.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.050+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7ffb941059c0 con 0x7ffb940fe230 2024-02-16T22:19:36.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.051+0000 7ffb917fa700 1 -- 172.21.15.2:0/1258714088 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7ffb88057e50 con 0x7ffb940fe230 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffb7c0692a0 msgr2=0x7ffb7c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffb7c0692a0 0x7ffb7c06b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb94065cb0 tx=0x7ffb8400b040 comp rx=0 tx=0).stop 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 msgr2=0x7ffb9410ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 0x7ffb9410ddf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ffb880093b0 tx=0x7ffb88009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 shutdown_connections 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7ffb7c0692a0 0x7ffb7c06b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffb940fb1a0 0x7ffb9410d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 --2- 172.21.15.2:0/1258714088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb940fe230 0x7ffb9410ddf0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 >> 172.21.15.2:0/1258714088 conn(0x7ffb940f6c00 msgr2=0x7ffb94101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:36.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 shutdown_connections 2024-02-16T22:19:36.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:36.053+0000 7ffb9b114700 1 -- 172.21.15.2:0/1258714088 wait complete. 2024-02-16T22:19:36.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:35 smithi103 ceph-mon[36259]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-02-16T22:19:36.769 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-16T22:19:37.031 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:38.158 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:37 smithi002 ceph-mon[29154]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-16T22:19:38.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:37 smithi103 ceph-mon[36259]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-16T22:19:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 -- 172.21.15.2:0/4023401481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 msgr2=0x7f85500fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 --2- 172.21.15.2:0/4023401481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 0x7f85500fc250 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8540004940 tx=0x7f8540009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 -- 172.21.15.2:0/4023401481 shutdown_connections 2024-02-16T22:19:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 --2- 172.21.15.2:0/4023401481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500fcfb0 0x7f85500fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:38.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 --2- 172.21.15.2:0/4023401481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 0x7f85500fc250 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 -- 172.21.15.2:0/4023401481 >> 172.21.15.2:0/4023401481 conn(0x7f85500f7520 msgr2=0x7f85500f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 -- 172.21.15.2:0/4023401481 shutdown_connections 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 -- 172.21.15.2:0/4023401481 wait complete. 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 Processor -- start 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 -- start start 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 0x7f8550110590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500fcfb0 0x7f8550112ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8550066d10 con 0x7f85500fbe40 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.687+0000 7f8555611700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8550066e50 con 0x7f85500fcfb0 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f854effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 0x7f8550110590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f854effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 0x7f8550110590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55336/0 (socket says 172.21.15.2:55336) 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f854effd700 1 -- 172.21.15.2:0/2226723701 learned_addr learned my addr 172.21.15.2:0/2226723701 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f8547fff700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500fcfb0 0x7f8550112ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:38.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f854effd700 1 -- 172.21.15.2:0/2226723701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500fcfb0 msgr2=0x7f8550112ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:38.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f854effd700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500fcfb0 0x7f8550112ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:38.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f854effd700 1 -- 172.21.15.2:0/2226723701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8540005040 con 0x7f85500fbe40 2024-02-16T22:19:38.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f854effd700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 0x7f8550110590 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f85400093b0 tx=0x7f8540009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:38.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f85477fe700 1 -- 172.21.15.2:0/2226723701 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f854000e040 con 0x7f85500fbe40 2024-02-16T22:19:38.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f85477fe700 1 -- 172.21.15.2:0/2226723701 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f854000bcc0 con 0x7f85500fbe40 2024-02-16T22:19:38.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85501130e0 con 0x7f85500fbe40 2024-02-16T22:19:38.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f85477fe700 1 -- 172.21.15.2:0/2226723701 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f854001c630 con 0x7f85500fbe40 2024-02-16T22:19:38.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.688+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85501135d0 con 0x7f85500fbe40 2024-02-16T22:19:38.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.689+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855004bed0 con 0x7f85500fbe40 2024-02-16T22:19:38.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.690+0000 7f85477fe700 1 -- 172.21.15.2:0/2226723701 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f8540013980 con 0x7f85500fbe40 2024-02-16T22:19:38.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.690+0000 7f85477fe700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f853c0692a0 0x7f853c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:38.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.690+0000 7f85477fe700 1 -- 172.21.15.2:0/2226723701 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8540088570 con 0x7f85500fbe40 2024-02-16T22:19:38.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.690+0000 7f8547fff700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f853c0692a0 0x7f853c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:38.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.691+0000 7f8547fff700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f853c0692a0 0x7f853c06b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8538009fd0 tx=0x7f853800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:38.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.694+0000 7f85477fe700 1 -- 172.21.15.2:0/2226723701 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8540054850 con 0x7f85500fbe40 2024-02-16T22:19:38.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.888+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f8550104930 con 0x7f85500fbe40 2024-02-16T22:19:38.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.888+0000 7f85477fe700 1 -- 172.21.15.2:0/2226723701 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f8540057e70 con 0x7f85500fbe40 2024-02-16T22:19:38.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f853c0692a0 msgr2=0x7f853c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:38.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f853c0692a0 0x7f853c06b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8538009fd0 tx=0x7f853800b040 comp rx=0 tx=0).stop 2024-02-16T22:19:38.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 msgr2=0x7f8550110590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:38.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 0x7f8550110590 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f85400093b0 tx=0x7f8540009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:38.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 shutdown_connections 2024-02-16T22:19:38.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f853c0692a0 0x7f853c06b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:38.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85500fbe40 0x7f8550110590 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:38.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 --2- 172.21.15.2:0/2226723701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85500fcfb0 0x7f8550112ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:38.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 >> 172.21.15.2:0/2226723701 conn(0x7f85500f7520 msgr2=0x7f8550100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:38.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 shutdown_connections 2024-02-16T22:19:38.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:38.890+0000 7f8555611700 1 -- 172.21.15.2:0/2226723701 wait complete. 2024-02-16T22:19:39.167 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:19:39.226 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:19:39.594 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-16T22:19:39.861 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:40.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:39 smithi103 ceph-mon[36259]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-16T22:19:40.138 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:39 smithi002 ceph-mon[29154]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-16T22:19:40.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:40 smithi002 ceph-mon[29154]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-16T22:19:41.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:40 smithi103 ceph-mon[36259]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.549+0000 7f70a9b50700 1 -- 172.21.15.2:0/3087050162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fc7f0 msgr2=0x7f70a40fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.549+0000 7f70a9b50700 1 --2- 172.21.15.2:0/3087050162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fc7f0 0x7f70a40fcc40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7098004940 tx=0x7f7098009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.549+0000 7f70a9b50700 1 -- 172.21.15.2:0/3087050162 shutdown_connections 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.549+0000 7f70a9b50700 1 --2- 172.21.15.2:0/3087050162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fc7f0 0x7f70a40fcc40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.549+0000 7f70a9b50700 1 --2- 172.21.15.2:0/3087050162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70a40fb5e0 0x7f70a40fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.549+0000 7f70a9b50700 1 -- 172.21.15.2:0/3087050162 >> 172.21.15.2:0/3087050162 conn(0x7f70a40f6ca0 msgr2=0x7f70a40f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.549+0000 7f70a9b50700 1 -- 172.21.15.2:0/3087050162 shutdown_connections 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.549+0000 7f70a9b50700 1 -- 172.21.15.2:0/3087050162 wait complete. 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a9b50700 1 Processor -- start 2024-02-16T22:19:41.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a9b50700 1 -- start start 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a9b50700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fb5e0 0x7f70a40ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a9b50700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70a40fc7f0 0x7f70a40fdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a9b50700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40fe130 con 0x7f70a40fb5e0 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a9b50700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40fe2a0 con 0x7f70a40fc7f0 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fb5e0 0x7f70a40ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a2ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70a40fc7f0 0x7f70a40fdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fb5e0 0x7f70a40ff6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54728/0 (socket says 172.21.15.2:54728) 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.550+0000 7f70a37fe700 1 -- 172.21.15.2:0/1624802429 learned_addr learned my addr 172.21.15.2:0/1624802429 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f70a2ffd700 1 -- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fb5e0 msgr2=0x7f70a40ff6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f70a2ffd700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fb5e0 0x7f70a40ff6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f70a2ffd700 1 -- 172.21.15.2:0/1624802429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7098005040 con 0x7f70a40fc7f0 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f70a37fe700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fb5e0 0x7f70a40ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-16T22:19:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f70a2ffd700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70a40fc7f0 0x7f70a40fdb30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7098000c00 tx=0x7f7098009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:41.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f708bfff700 1 -- 172.21.15.2:0/1624802429 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f709800e040 con 0x7f70a40fc7f0 2024-02-16T22:19:41.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a40fe520 con 0x7f70a40fc7f0 2024-02-16T22:19:41.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f708bfff700 1 -- 172.21.15.2:0/1624802429 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f709800bbf0 con 0x7f70a40fc7f0 2024-02-16T22:19:41.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f708bfff700 1 -- 172.21.15.2:0/1624802429 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f709801c3f0 con 0x7f70a40fc7f0 2024-02-16T22:19:41.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.551+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a4115990 con 0x7f70a40fc7f0 2024-02-16T22:19:41.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.552+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a4107890 con 0x7f70a40fc7f0 2024-02-16T22:19:41.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.552+0000 7f708bfff700 1 -- 172.21.15.2:0/1624802429 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f709800bd60 con 0x7f70a40fc7f0 2024-02-16T22:19:41.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.553+0000 7f708bfff700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f708c0692a0 0x7f708c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:41.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.553+0000 7f708bfff700 1 -- 172.21.15.2:0/1624802429 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7098087da0 con 0x7f70a40fc7f0 2024-02-16T22:19:41.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.553+0000 7f70a37fe700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f708c0692a0 0x7f708c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:41.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.553+0000 7f70a37fe700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f708c0692a0 0x7f708c06b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f709400afd0 tx=0x7f7094009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:41.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.557+0000 7f708bfff700 1 -- 172.21.15.2:0/1624802429 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7098057bc0 con 0x7f70a40fc7f0 2024-02-16T22:19:41.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.758+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f70a404bed0 con 0x7f70a40fc7f0 2024-02-16T22:19:41.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.759+0000 7f708bfff700 1 -- 172.21.15.2:0/1624802429 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f7098057750 con 0x7f70a40fc7f0 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f708c0692a0 msgr2=0x7f708c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f708c0692a0 0x7f708c06b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f709400afd0 tx=0x7f7094009040 comp rx=0 tx=0).stop 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70a40fc7f0 msgr2=0x7f70a40fdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70a40fc7f0 0x7f70a40fdb30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7098000c00 tx=0x7f7098009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 shutdown_connections 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f708c0692a0 0x7f708c06b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70a40fb5e0 0x7f70a40ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 --2- 172.21.15.2:0/1624802429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70a40fc7f0 0x7f70a40fdb30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 >> 172.21.15.2:0/1624802429 conn(0x7f70a40f6ca0 msgr2=0x7f70a40ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 shutdown_connections 2024-02-16T22:19:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:41.761+0000 7f70a9b50700 1 -- 172.21.15.2:0/1624802429 wait complete. 2024-02-16T22:19:42.527 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-16T22:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:43.934 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:43 smithi002 ceph-mon[29154]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-16T22:19:44.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:43 smithi103 ceph-mon[36259]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-16T22:19:44.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 -- 172.21.15.2:0/2747144716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fcfb0 msgr2=0x7f2a280fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:44.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 --2- 172.21.15.2:0/2747144716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fcfb0 0x7f2a280fd420 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c004940 tx=0x7f2a1c009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:44.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 -- 172.21.15.2:0/2747144716 shutdown_connections 2024-02-16T22:19:44.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 --2- 172.21.15.2:0/2747144716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fcfb0 0x7f2a280fd420 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 --2- 172.21.15.2:0/2747144716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a280fbe40 0x7f2a280fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 -- 172.21.15.2:0/2747144716 >> 172.21.15.2:0/2747144716 conn(0x7f2a280f7520 msgr2=0x7f2a280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 -- 172.21.15.2:0/2747144716 shutdown_connections 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 -- 172.21.15.2:0/2747144716 wait complete. 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 Processor -- start 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.581+0000 7f2a2c977700 1 -- start start 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2c977700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fbe40 0x7f2a2810d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2c977700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a280fcfb0 0x7f2a2810ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2c977700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2810e3f0 con 0x7f2a280fbe40 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2c977700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2810ec90 con 0x7f2a280fcfb0 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fbe40 0x7f2a2810d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fbe40 0x7f2a2810d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54746/0 (socket says 172.21.15.2:54746) 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2659c700 1 -- 172.21.15.2:0/3733083370 learned_addr learned my addr 172.21.15.2:0/3733083370 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:44.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a25d9b700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a280fcfb0 0x7f2a2810ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2659c700 1 -- 172.21.15.2:0/3733083370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a280fcfb0 msgr2=0x7f2a2810ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2659c700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a280fcfb0 0x7f2a2810ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2659c700 1 -- 172.21.15.2:0/3733083370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c005040 con 0x7f2a280fbe40 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a25d9b700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a280fcfb0 0x7f2a2810ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2659c700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fbe40 0x7f2a2810d8b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800eb40 tx=0x7f2a1800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a16ffd700 1 -- 172.21.15.2:0/3733083370 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a18007cc0 con 0x7f2a280fbe40 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2810ef10 con 0x7f2a280fbe40 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a16ffd700 1 -- 172.21.15.2:0/3733083370 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a1800f040 con 0x7f2a280fbe40 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a16ffd700 1 -- 172.21.15.2:0/3733083370 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a1800ca10 con 0x7f2a280fbe40 2024-02-16T22:19:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.582+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2810f450 con 0x7f2a280fbe40 2024-02-16T22:19:44.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.583+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2804bed0 con 0x7f2a280fbe40 2024-02-16T22:19:44.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.584+0000 7f2a16ffd700 1 -- 172.21.15.2:0/3733083370 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2a1800cc80 con 0x7f2a280fbe40 2024-02-16T22:19:44.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.584+0000 7f2a16ffd700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2a100692b0 0x7f2a1006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:44.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.584+0000 7f2a16ffd700 1 -- 172.21.15.2:0/3733083370 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2a18086dd0 con 0x7f2a280fbe40 2024-02-16T22:19:44.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.584+0000 7f2a25d9b700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2a100692b0 0x7f2a1006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:44.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.585+0000 7f2a25d9b700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2a100692b0 0x7f2a1006b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c013740 tx=0x7f2a1c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:44.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.589+0000 7f2a16ffd700 1 -- 172.21.15.2:0/3733083370 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2a1804dbd0 con 0x7f2a280fbe40 2024-02-16T22:19:44.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.799+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a28066660 con 0x7f2a100692b0 2024-02-16T22:19:44.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.805+0000 7f2a16ffd700 1 -- 172.21.15.2:0/3733083370 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f2a28066660 con 0x7f2a100692b0 2024-02-16T22:19:44.808 INFO:teuthology.orchestra.run.smithi002.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:19:44.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2a100692b0 msgr2=0x7f2a1006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:44.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2a100692b0 0x7f2a1006b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c013740 tx=0x7f2a1c009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:44.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fbe40 msgr2=0x7f2a2810d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:44.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fbe40 0x7f2a2810d8b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800eb40 tx=0x7f2a1800b040 comp rx=0 tx=0).stop 2024-02-16T22:19:44.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 shutdown_connections 2024-02-16T22:19:44.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f2a100692b0 0x7f2a1006b760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:44.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a280fbe40 0x7f2a2810d8b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:44.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 --2- 172.21.15.2:0/3733083370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a280fcfb0 0x7f2a2810ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:44.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 >> 172.21.15.2:0/3733083370 conn(0x7f2a280f7520 msgr2=0x7f2a28065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:44.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 shutdown_connections 2024-02-16T22:19:44.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:44.807+0000 7f2a2c977700 1 -- 172.21.15.2:0/3733083370 wait complete. 2024-02-16T22:19:45.602 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-16T22:19:45.602 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:19:45.602 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-16T22:19:45.899 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:19:45.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:45 smithi002 ceph-mon[29154]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-16T22:19:45.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:45 smithi002 ceph-mon[29154]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:45.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:45 smithi002 ceph-mon[29154]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:19:45.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:45.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:45 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:19:46.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:45 smithi103 ceph-mon[36259]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-16T22:19:46.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:45 smithi103 ceph-mon[36259]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:46.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:45 smithi103 ceph-mon[36259]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:19:46.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:46.076 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:45 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:19:47.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:46 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:19:47.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:46 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:19:47.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:46 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:47.203 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:46 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-16T22:19:47.204 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:46 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-16T22:19:47.204 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:46 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:19:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.761+0000 7fdef52f1700 1 -- 172.21.15.2:0/3782440357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdef00fcfb0 msgr2=0x7fdef00fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.761+0000 7fdef52f1700 1 --2- 172.21.15.2:0/3782440357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdef00fcfb0 0x7fdef00fd420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdee4004940 tx=0x7fdee4009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.761+0000 7fdef52f1700 1 -- 172.21.15.2:0/3782440357 shutdown_connections 2024-02-16T22:19:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.761+0000 7fdef52f1700 1 --2- 172.21.15.2:0/3782440357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdef00fcfb0 0x7fdef00fd420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.761+0000 7fdef52f1700 1 --2- 172.21.15.2:0/3782440357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef00fbe40 0x7fdef00fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.761+0000 7fdef52f1700 1 -- 172.21.15.2:0/3782440357 >> 172.21.15.2:0/3782440357 conn(0x7fdef00f74e0 msgr2=0x7fdef00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.761+0000 7fdef52f1700 1 -- 172.21.15.2:0/3782440357 shutdown_connections 2024-02-16T22:19:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.761+0000 7fdef52f1700 1 -- 172.21.15.2:0/3782440357 wait complete. 2024-02-16T22:19:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdef52f1700 1 Processor -- start 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdef52f1700 1 -- start start 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdef52f1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef00fbe40 0x7fdef00fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdef52f1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdef00fcfb0 0x7fdef00fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdef52f1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00fe980 con 0x7fdef00fbe40 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdef52f1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00feaf0 con 0x7fdef00fcfb0 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdeeeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef00fbe40 0x7fdef00fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdeeeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef00fbe40 0x7fdef00fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54756/0 (socket says 172.21.15.2:54756) 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdeeeffd700 1 -- 172.21.15.2:0/645107680 learned_addr learned my addr 172.21.15.2:0/645107680 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdeee7fc700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdef00fcfb0 0x7fdef00fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdeeeffd700 1 -- 172.21.15.2:0/645107680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdef00fcfb0 msgr2=0x7fdef00fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdeeeffd700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdef00fcfb0 0x7fdef00fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.762+0000 7fdeeeffd700 1 -- 172.21.15.2:0/645107680 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee4005040 con 0x7fdef00fbe40 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.763+0000 7fdeeeffd700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef00fbe40 0x7fdef00fff00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdee000eb40 tx=0x7fdee000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:47.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.763+0000 7fded77fe700 1 -- 172.21.15.2:0/645107680 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdee0007cc0 con 0x7fdef00fbe40 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.763+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef00fed70 con 0x7fdef00fbe40 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.763+0000 7fded77fe700 1 -- 172.21.15.2:0/645107680 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdee000f040 con 0x7fdef00fbe40 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.763+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef01158e0 con 0x7fdef00fbe40 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.763+0000 7fded77fe700 1 -- 172.21.15.2:0/645107680 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdee000ca10 con 0x7fdef00fbe40 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.764+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef004bed0 con 0x7fdef00fbe40 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.765+0000 7fded77fe700 1 -- 172.21.15.2:0/645107680 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fdee000cc80 con 0x7fdef00fbe40 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.765+0000 7fded77fe700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fded80692a0 0x7fded806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.765+0000 7fded77fe700 1 -- 172.21.15.2:0/645107680 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdee0086df0 con 0x7fdef00fbe40 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.765+0000 7fdeee7fc700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fded80692a0 0x7fded806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.766+0000 7fdeee7fc700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fded80692a0 0x7fded806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009b30 tx=0x7fdee4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:47.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.773+0000 7fded77fe700 1 -- 172.21.15.2:0/645107680 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdee004dbd0 con 0x7fdef00fbe40 2024-02-16T22:19:47.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.989+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdef0065dd0 con 0x7fded80692a0 2024-02-16T22:19:47.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.990+0000 7fded77fe700 1 -- 172.21.15.2:0/645107680 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fdef0065dd0 con 0x7fded80692a0 2024-02-16T22:19:47.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fded80692a0 msgr2=0x7fded806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:47.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fded80692a0 0x7fded806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009b30 tx=0x7fdee4009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:47.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef00fbe40 msgr2=0x7fdef00fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:47.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef00fbe40 0x7fdef00fff00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdee000eb40 tx=0x7fdee000b040 comp rx=0 tx=0).stop 2024-02-16T22:19:47.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 shutdown_connections 2024-02-16T22:19:47.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fded80692a0 0x7fded806b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:47.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef00fbe40 0x7fdef00fff00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:47.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 --2- 172.21.15.2:0/645107680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdef00fcfb0 0x7fdef00fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:47.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 >> 172.21.15.2:0/645107680 conn(0x7fdef00f74e0 msgr2=0x7fdef00656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:47.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 shutdown_connections 2024-02-16T22:19:47.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:47.992+0000 7fdef52f1700 1 -- 172.21.15.2:0/645107680 wait complete. 2024-02-16T22:19:48.005 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-16T22:19:48.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.105+0000 7f94eb180700 1 -- 172.21.15.2:0/1862612079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 msgr2=0x7f94e40fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.105+0000 7f94eb180700 1 --2- 172.21.15.2:0/1862612079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 0x7f94e40fc260 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f94d8004940 tx=0x7f94d8009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:48.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.105+0000 7f94eb180700 1 -- 172.21.15.2:0/1862612079 shutdown_connections 2024-02-16T22:19:48.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.105+0000 7f94eb180700 1 --2- 172.21.15.2:0/1862612079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e40fcfc0 0x7f94e40fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.105+0000 7f94eb180700 1 --2- 172.21.15.2:0/1862612079 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 0x7f94e40fc260 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.105+0000 7f94eb180700 1 -- 172.21.15.2:0/1862612079 >> 172.21.15.2:0/1862612079 conn(0x7f94e40f7510 msgr2=0x7f94e40f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:48.107 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:47 smithi002 ceph-mon[29154]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:19:48.108 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:47 smithi002 ceph-mon[29154]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-16T22:19:48.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.105+0000 7f94eb180700 1 -- 172.21.15.2:0/1862612079 shutdown_connections 2024-02-16T22:19:48.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.105+0000 7f94eb180700 1 -- 172.21.15.2:0/1862612079 wait complete. 2024-02-16T22:19:48.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.106+0000 7f94eb180700 1 Processor -- start 2024-02-16T22:19:48.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.106+0000 7f94eb180700 1 -- start start 2024-02-16T22:19:48.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.106+0000 7f94eb180700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 0x7f94e410c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94eb180700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e40fcfc0 0x7f94e410cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94eb180700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e410d390 con 0x7f94e40fbe50 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e8f1c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 0x7f94e410c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e8f1c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 0x7f94e410c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54770/0 (socket says 172.21.15.2:54770) 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e8f1c700 1 -- 172.21.15.2:0/3667598273 learned_addr learned my addr 172.21.15.2:0/3667598273 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e410dc30 con 0x7f94e40fcfc0 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e3fff700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e40fcfc0 0x7f94e410cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e8f1c700 1 -- 172.21.15.2:0/3667598273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e40fcfc0 msgr2=0x7f94e410cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e8f1c700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e40fcfc0 0x7f94e410cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e8f1c700 1 -- 172.21.15.2:0/3667598273 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d8005040 con 0x7f94e40fbe50 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e3fff700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e40fcfc0 0x7f94e410cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e8f1c700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 0x7f94e410c850 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f94d80093b0 tx=0x7f94d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.107+0000 7f94e17fa700 1 -- 172.21.15.2:0/3667598273 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94d800e040 con 0x7f94e40fbe50 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.108+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e410deb0 con 0x7f94e40fbe50 2024-02-16T22:19:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.108+0000 7f94e17fa700 1 -- 172.21.15.2:0/3667598273 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94d800bcc0 con 0x7f94e40fbe50 2024-02-16T22:19:48.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.108+0000 7f94e17fa700 1 -- 172.21.15.2:0/3667598273 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94d801c630 con 0x7f94e40fbe50 2024-02-16T22:19:48.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.108+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e410e3f0 con 0x7f94e40fbe50 2024-02-16T22:19:48.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.109+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e404bed0 con 0x7f94e40fbe50 2024-02-16T22:19:48.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.109+0000 7f94e17fa700 1 -- 172.21.15.2:0/3667598273 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f94d8013980 con 0x7f94e40fbe50 2024-02-16T22:19:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.110+0000 7f94e17fa700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f94d406b4b0 0x7f94d406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.110+0000 7f94e17fa700 1 -- 172.21.15.2:0/3667598273 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f94d8088480 con 0x7f94e40fbe50 2024-02-16T22:19:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.110+0000 7f94e3fff700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f94d406b4b0 0x7f94d406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.110+0000 7f94e3fff700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f94d406b4b0 0x7f94d406d960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f94d0010520 tx=0x7f94d000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.114+0000 7f94e17fa700 1 -- 172.21.15.2:0/3667598273 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f94d8058290 con 0x7f94e40fbe50 2024-02-16T22:19:48.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:47 smithi103 ceph-mon[36259]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-16T22:19:48.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:47 smithi103 ceph-mon[36259]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-16T22:19:48.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.333+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f94e410e040 con 0x7f94d406b4b0 2024-02-16T22:19:48.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.334+0000 7f94e17fa700 1 -- 172.21.15.2:0/3667598273 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f94e410e040 con 0x7f94d406b4b0 2024-02-16T22:19:48.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.336+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f94d406b4b0 msgr2=0x7f94d406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.336+0000 7f94eb180700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f94d406b4b0 0x7f94d406d960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f94d0010520 tx=0x7f94d000b040 comp rx=0 tx=0).stop 2024-02-16T22:19:48.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.336+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 msgr2=0x7f94e410c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.336+0000 7f94eb180700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 0x7f94e410c850 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f94d80093b0 tx=0x7f94d8009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:48.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.336+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 shutdown_connections 2024-02-16T22:19:48.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.336+0000 7f94eb180700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f94d406b4b0 0x7f94d406d960 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.336+0000 7f94eb180700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94e40fbe50 0x7f94e410c850 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.336+0000 7f94eb180700 1 --2- 172.21.15.2:0/3667598273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94e40fcfc0 0x7f94e410cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.336+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 >> 172.21.15.2:0/3667598273 conn(0x7f94e40f7510 msgr2=0x7f94e4100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:48.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.337+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 shutdown_connections 2024-02-16T22:19:48.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.337+0000 7f94eb180700 1 -- 172.21.15.2:0/3667598273 wait complete. 2024-02-16T22:19:48.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.444+0000 7fc52dc68700 1 -- 172.21.15.2:0/2930794524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 msgr2=0x7fc5280ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.444+0000 7fc52dc68700 1 --2- 172.21.15.2:0/2930794524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 0x7fc5280ffe60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc518004940 tx=0x7fc518009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.444+0000 7fc52dc68700 1 -- 172.21.15.2:0/2930794524 shutdown_connections 2024-02-16T22:19:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.444+0000 7fc52dc68700 1 --2- 172.21.15.2:0/2930794524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5281003a0 0x7fc528102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.444+0000 7fc52dc68700 1 --2- 172.21.15.2:0/2930794524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 0x7fc5280ffe60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.444+0000 7fc52dc68700 1 -- 172.21.15.2:0/2930794524 >> 172.21.15.2:0/2930794524 conn(0x7fc5280f7500 msgr2=0x7fc5280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.444+0000 7fc52dc68700 1 -- 172.21.15.2:0/2930794524 shutdown_connections 2024-02-16T22:19:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.444+0000 7fc52dc68700 1 -- 172.21.15.2:0/2930794524 wait complete. 2024-02-16T22:19:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.444+0000 7fc52dc68700 1 Processor -- start 2024-02-16T22:19:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc52dc68700 1 -- start start 2024-02-16T22:19:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc52dc68700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 0x7fc52810fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc52dc68700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5281003a0 0x7fc528110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc52dc68700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc528110650 con 0x7fc5280fda80 2024-02-16T22:19:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc52dc68700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc528110ef0 con 0x7fc5281003a0 2024-02-16T22:19:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc5277fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 0x7fc52810fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc5277fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 0x7fc52810fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54788/0 (socket says 172.21.15.2:54788) 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc5277fe700 1 -- 172.21.15.2:0/1787530369 learned_addr learned my addr 172.21.15.2:0/1787530369 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc526ffd700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5281003a0 0x7fc528110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc5277fe700 1 -- 172.21.15.2:0/1787530369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5281003a0 msgr2=0x7fc528110050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc5277fe700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5281003a0 0x7fc528110050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc5277fe700 1 -- 172.21.15.2:0/1787530369 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc518005040 con 0x7fc5280fda80 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc5277fe700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 0x7fc52810fb10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc5180093b0 tx=0x7fc518009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc50ffff700 1 -- 172.21.15.2:0/1787530369 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc51800e040 con 0x7fc5280fda80 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.445+0000 7fc50ffff700 1 -- 172.21.15.2:0/1787530369 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc51800bcc0 con 0x7fc5280fda80 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.446+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc528111170 con 0x7fc5280fda80 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.446+0000 7fc50ffff700 1 -- 172.21.15.2:0/1787530369 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc51801c630 con 0x7fc5280fda80 2024-02-16T22:19:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.446+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5281116b0 con 0x7fc5280fda80 2024-02-16T22:19:48.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.447+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52804bed0 con 0x7fc5280fda80 2024-02-16T22:19:48.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.447+0000 7fc50ffff700 1 -- 172.21.15.2:0/1787530369 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fc5180139f0 con 0x7fc5280fda80 2024-02-16T22:19:48.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.448+0000 7fc50ffff700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fc5100692a0 0x7fc51006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:48.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.448+0000 7fc526ffd700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fc5100692a0 0x7fc51006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:48.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.448+0000 7fc50ffff700 1 -- 172.21.15.2:0/1787530369 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc518088630 con 0x7fc5280fda80 2024-02-16T22:19:48.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.448+0000 7fc526ffd700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fc5100692a0 0x7fc51006b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc51c007fd0 tx=0x7fc51c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.452+0000 7fc50ffff700 1 -- 172.21.15.2:0/1787530369 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc518058450 con 0x7fc5280fda80 2024-02-16T22:19:48.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.663+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5280fbf80 con 0x7fc5100692a0 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.673+0000 7fc50ffff700 1 -- 172.21.15.2:0/1787530369 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fc5280fbf80 con 0x7fc5100692a0 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.smithi002 smithi002 *:9093,9094 running (4m) 26s ago 6m 25.0M - 0.25.0 c8568f914cd2 2b7780bcc981 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi002 smithi002 running (6m) 26s ago 6m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 c1d597f23cdd 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi103 smithi103 running (5m) 31s ago 5m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 db2a59e09a51 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.smithi002 smithi002 *:3000 running (4m) 26s ago 6m 87.3M - 9.4.7 954c08fa6188 14099b699363 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi002.hhufhl smithi002 running (44s) 26s ago 44s 15.3M - 17.2.7-513-gb88fae66 870a0944ccb8 12d8ab360b57 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi002.llkqse smithi002 running (52s) 26s ago 52s 10.5M - 17.2.7-513-gb88fae66 870a0944ccb8 8e266571411c 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi103.epsgnr smithi103 running (42s) 31s ago 41s 14.5M - 17.2.7-513-gb88fae66 870a0944ccb8 d43d0b5d182c 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi103.kbizjo smithi103 running (50s) 31s ago 49s 14.6M - 17.2.7-513-gb88fae66 870a0944ccb8 86d4a2e35372 2024-02-16T22:19:48.675 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi002.resmjv smithi002 *:9283 running (7m) 26s ago 7m 480M - 17.2.7-513-gb88fae66 870a0944ccb8 f9b132a0aa69 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi103.ydydbo smithi103 *:8443,9283 running (5m) 31s ago 5m 433M - 17.2.7-513-gb88fae66 870a0944ccb8 463897d4efc5 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi002 smithi002 running (7m) 26s ago 7m 45.2M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 cc6b0b4bb587 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi103 smithi103 running (5m) 31s ago 5m 35.6M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 3d198bee2a92 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi002 smithi002 *:9100 running (6m) 26s ago 6m 15.9M - 1.5.0 0da6a335fe13 e6146a57c445 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi103 smithi103 *:9100 running (5m) 31s ago 5m 17.0M - 1.5.0 0da6a335fe13 0b21b215b785 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (4m) 26s ago 4m 48.7M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 a3f19d068baf 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (4m) 26s ago 4m 46.1M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 58697cab97b0 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (3m) 26s ago 3m 47.1M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 634798c2dbad 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi103 running (3m) 31s ago 3m 43.9M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 a5332c698672 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi103 running (2m) 31s ago 2m 42.5M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 fa2a12f4b710 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi103 running (2m) 31s ago 2m 42.1M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 175f4eefd1f6 2024-02-16T22:19:48.676 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.smithi002 smithi002 *:9095 running (4m) 26s ago 5m 55.1M - 2.43.0 a07b618ecd1d d03f44725127 2024-02-16T22:19:48.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fc5100692a0 msgr2=0x7fc51006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fc5100692a0 0x7fc51006b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc51c007fd0 tx=0x7fc51c009040 comp rx=0 tx=0).stop 2024-02-16T22:19:48.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 msgr2=0x7fc52810fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 0x7fc52810fb10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc5180093b0 tx=0x7fc518009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 shutdown_connections 2024-02-16T22:19:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fc5100692a0 0x7fc51006b750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5280fda80 0x7fc52810fb10 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 --2- 172.21.15.2:0/1787530369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5281003a0 0x7fc528110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 >> 172.21.15.2:0/1787530369 conn(0x7fc5280f7500 msgr2=0x7fc5280fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 shutdown_connections 2024-02-16T22:19:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.675+0000 7fc52dc68700 1 -- 172.21.15.2:0/1787530369 wait complete. 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.779+0000 7f03629e8700 1 -- 172.21.15.2:0/353461980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 msgr2=0x7f035c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.779+0000 7f03629e8700 1 --2- 172.21.15.2:0/353461980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 0x7f035c0fd420 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0350004940 tx=0x7f0350009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.779+0000 7f03629e8700 1 -- 172.21.15.2:0/353461980 shutdown_connections 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.779+0000 7f03629e8700 1 --2- 172.21.15.2:0/353461980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 0x7f035c0fd420 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.779+0000 7f03629e8700 1 --2- 172.21.15.2:0/353461980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035c0fbe40 0x7f035c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.779+0000 7f03629e8700 1 -- 172.21.15.2:0/353461980 >> 172.21.15.2:0/353461980 conn(0x7f035c0f7500 msgr2=0x7f035c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f03629e8700 1 -- 172.21.15.2:0/353461980 shutdown_connections 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f03629e8700 1 -- 172.21.15.2:0/353461980 wait complete. 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f03629e8700 1 Processor -- start 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f03629e8700 1 -- start start 2024-02-16T22:19:48.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f03629e8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035c0fbe40 0x7f035c110690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f03629e8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 0x7f035c112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f03629e8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c066d10 con 0x7f035c0fcfb0 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f03629e8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c066e50 con 0x7f035c0fbe40 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f035b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 0x7f035c112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f035b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 0x7f035c112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54814/0 (socket says 172.21.15.2:54814) 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f035bfff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035c0fbe40 0x7f035c110690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.780+0000 7f035b7fe700 1 -- 172.21.15.2:0/1129552929 learned_addr learned my addr 172.21.15.2:0/1129552929 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.781+0000 7f035b7fe700 1 -- 172.21.15.2:0/1129552929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035c0fbe40 msgr2=0x7f035c110690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.781+0000 7f035b7fe700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035c0fbe40 0x7f035c110690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.781+0000 7f035b7fe700 1 -- 172.21.15.2:0/1129552929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0350005040 con 0x7f035c0fcfb0 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.781+0000 7f035b7fe700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 0x7f035c112be0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f03500093b0 tx=0x7f0350009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.781+0000 7f0358ff9700 1 -- 172.21.15.2:0/1129552929 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f035000e040 con 0x7f035c0fcfb0 2024-02-16T22:19:48.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.781+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035c1131e0 con 0x7f035c0fcfb0 2024-02-16T22:19:48.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.781+0000 7f0358ff9700 1 -- 172.21.15.2:0/1129552929 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f035000bcc0 con 0x7f035c0fcfb0 2024-02-16T22:19:48.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.781+0000 7f0358ff9700 1 -- 172.21.15.2:0/1129552929 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f035001c630 con 0x7f035c0fcfb0 2024-02-16T22:19:48.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.781+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035c1136d0 con 0x7f035c0fcfb0 2024-02-16T22:19:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.782+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035c04bed0 con 0x7f035c0fcfb0 2024-02-16T22:19:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.783+0000 7f0358ff9700 1 -- 172.21.15.2:0/1129552929 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f0350013980 con 0x7f035c0fcfb0 2024-02-16T22:19:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.783+0000 7f0358ff9700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f03440692a0 0x7f034406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.783+0000 7f0358ff9700 1 -- 172.21.15.2:0/1129552929 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f03500884a0 con 0x7f035c0fcfb0 2024-02-16T22:19:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.783+0000 7f035bfff700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f03440692a0 0x7f034406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.783+0000 7f035bfff700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f03440692a0 0x7f034406b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f035c066990 tx=0x7f034c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:48.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:48.787+0000 7f0358ff9700 1 -- 172.21.15.2:0/1129552929 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0350054820 con 0x7f035c0fcfb0 2024-02-16T22:19:49.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.080+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f035c104950 con 0x7f035c0fcfb0 2024-02-16T22:19:49.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.081+0000 7f0358ff9700 1 -- 172.21.15.2:0/1129552929 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f0350057e40 con 0x7f035c0fcfb0 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 2 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 2 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 6 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": { 2024-02-16T22:19:49.083 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 4 2024-02-16T22:19:49.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:19:49.084 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-16T22:19:49.084 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 14 2024-02-16T22:19:49.084 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-16T22:19:49.084 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-16T22:19:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f03440692a0 msgr2=0x7f034406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f03440692a0 0x7f034406b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f035c066990 tx=0x7f034c00b040 comp rx=0 tx=0).stop 2024-02-16T22:19:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 msgr2=0x7f035c112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 0x7f035c112be0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f03500093b0 tx=0x7f0350009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 shutdown_connections 2024-02-16T22:19:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f03440692a0 0x7f034406b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f035c0fbe40 0x7f035c110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 --2- 172.21.15.2:0/1129552929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f035c0fcfb0 0x7f035c112be0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 >> 172.21.15.2:0/1129552929 conn(0x7f035c0f7500 msgr2=0x7f035c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:49.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 shutdown_connections 2024-02-16T22:19:49.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.083+0000 7f03629e8700 1 -- 172.21.15.2:0/1129552929 wait complete. 2024-02-16T22:19:49.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.188+0000 7fed75a7a700 1 -- 172.21.15.2:0/319412912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 msgr2=0x7fed700fb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.188+0000 7fed75a7a700 1 --2- 172.21.15.2:0/319412912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 0x7fed700fb710 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fed60004940 tx=0x7fed60009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:49.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.188+0000 7fed75a7a700 1 -- 172.21.15.2:0/319412912 shutdown_connections 2024-02-16T22:19:49.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.188+0000 7fed75a7a700 1 --2- 172.21.15.2:0/319412912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed700fbce0 0x7fed70102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.188+0000 7fed75a7a700 1 --2- 172.21.15.2:0/319412912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 0x7fed700fb710 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.188+0000 7fed75a7a700 1 -- 172.21.15.2:0/319412912 >> 172.21.15.2:0/319412912 conn(0x7fed700f6ca0 msgr2=0x7fed700f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.188+0000 7fed75a7a700 1 -- 172.21.15.2:0/319412912 shutdown_connections 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.188+0000 7fed75a7a700 1 -- 172.21.15.2:0/319412912 wait complete. 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed75a7a700 1 Processor -- start 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed75a7a700 1 -- start start 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed75a7a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 0x7fed700fde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed75a7a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed700fc2e0 0x7fed700fc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed75a7a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed700fcc90 con 0x7fed700fb300 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed75a7a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed700fcdd0 con 0x7fed700fc2e0 2024-02-16T22:19:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 0x7fed700fde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 0x7fed700fde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54830/0 (socket says 172.21.15.2:54830) 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed6f7fe700 1 -- 172.21.15.2:0/3664765214 learned_addr learned my addr 172.21.15.2:0/3664765214 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.189+0000 7fed6effd700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed700fc2e0 0x7fed700fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.190+0000 7fed6f7fe700 1 -- 172.21.15.2:0/3664765214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed700fc2e0 msgr2=0x7fed700fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.190+0000 7fed6f7fe700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed700fc2e0 0x7fed700fc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.190+0000 7fed6f7fe700 1 -- 172.21.15.2:0/3664765214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed60005040 con 0x7fed700fb300 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.190+0000 7fed6f7fe700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 0x7fed700fde10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fed6000b3b0 tx=0x7fed60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.190+0000 7fed57fff700 1 -- 172.21.15.2:0/3664765214 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed6000e040 con 0x7fed700fb300 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.190+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed700fd050 con 0x7fed700fb300 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.190+0000 7fed57fff700 1 -- 172.21.15.2:0/3664765214 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed6000bbf0 con 0x7fed700fb300 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.190+0000 7fed57fff700 1 -- 172.21.15.2:0/3664765214 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed6001c3f0 con 0x7fed700fb300 2024-02-16T22:19:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.190+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed70115d70 con 0x7fed700fb300 2024-02-16T22:19:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.191+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed70107ab0 con 0x7fed700fb300 2024-02-16T22:19:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.191+0000 7fed57fff700 1 -- 172.21.15.2:0/3664765214 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fed6000bd60 con 0x7fed700fb300 2024-02-16T22:19:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.192+0000 7fed57fff700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fed580692a0 0x7fed5806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.192+0000 7fed57fff700 1 -- 172.21.15.2:0/3664765214 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fed60088550 con 0x7fed700fb300 2024-02-16T22:19:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.192+0000 7fed6effd700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fed580692a0 0x7fed5806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.192+0000 7fed6effd700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fed580692a0 0x7fed5806b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fed700fe490 tx=0x7fed6400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:49.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.196+0000 7fed57fff700 1 -- 172.21.15.2:0/3664765214 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fed60054830 con 0x7fed700fb300 2024-02-16T22:19:49.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.438+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fed70105a70 con 0x7fed700fb300 2024-02-16T22:19:49.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.438+0000 7fed57fff700 1 -- 172.21.15.2:0/3664765214 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1830 (secure 0 0 0) 0x7fed60057e50 con 0x7fed700fb300 2024-02-16T22:19:49.440 INFO:teuthology.orchestra.run.smithi002.stdout:e16 2024-02-16T22:19:49.440 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-16T22:19:49.440 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 16 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-02-16T22:18:52.996722+0000 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2024-02-16T22:19:10.960547+0000 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 60 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2024-02-16T22:19:49.441 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 46 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 1 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:in 0 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24263} 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3] 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2024-02-16T22:19:49.442 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data enabled 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 1 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.kbizjo{0:24263} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.hhufhl{0:14486} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.2:6828/971745533,v1:172.21.15.2:6829/971745533] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.llkqse{-1:14496} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.2:6826/3769723051,v1:172.21.15.2:6827/3769723051] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:19:49.443 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.epsgnr{-1:24271} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.103:6826/1310923282,v1:172.21.15.103:6827/1310923282] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:19:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.441+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fed580692a0 msgr2=0x7fed5806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.441+0000 7fed75a7a700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fed580692a0 0x7fed5806b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fed700fe490 tx=0x7fed6400b040 comp rx=0 tx=0).stop 2024-02-16T22:19:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.441+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 msgr2=0x7fed700fde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.441+0000 7fed75a7a700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 0x7fed700fde10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fed6000b3b0 tx=0x7fed60009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.441+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 shutdown_connections 2024-02-16T22:19:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.441+0000 7fed75a7a700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fed580692a0 0x7fed5806b750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.441+0000 7fed75a7a700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed700fb300 0x7fed700fde10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.441+0000 7fed75a7a700 1 --2- 172.21.15.2:0/3664765214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed700fc2e0 0x7fed700fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.441+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 >> 172.21.15.2:0/3664765214 conn(0x7fed700f6ca0 msgr2=0x7fed70100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.442+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 shutdown_connections 2024-02-16T22:19:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.442+0000 7fed75a7a700 1 -- 172.21.15.2:0/3664765214 wait complete. 2024-02-16T22:19:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 16 2024-02-16T22:19:49.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.547+0000 7fbfe0899700 1 -- 172.21.15.2:0/2090092827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fcef0 msgr2=0x7fbfdc0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.547+0000 7fbfe0899700 1 --2- 172.21.15.2:0/2090092827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fcef0 0x7fbfdc0fd320 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc004940 tx=0x7fbfcc009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:49.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 -- 172.21.15.2:0/2090092827 shutdown_connections 2024-02-16T22:19:49.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 --2- 172.21.15.2:0/2090092827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfdc0fe0e0 0x7fbfdc0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 --2- 172.21.15.2:0/2090092827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fcef0 0x7fbfdc0fd320 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 -- 172.21.15.2:0/2090092827 >> 172.21.15.2:0/2090092827 conn(0x7fbfdc0f8620 msgr2=0x7fbfdc0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:49.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 -- 172.21.15.2:0/2090092827 shutdown_connections 2024-02-16T22:19:49.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 -- 172.21.15.2:0/2090092827 wait complete. 2024-02-16T22:19:49.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 Processor -- start 2024-02-16T22:19:49.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 -- start start 2024-02-16T22:19:49.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfdc0fcef0 0x7fbfdc071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:49.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fe0e0 0x7fbfdc0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0708e0 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfe0899700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc070a50 con 0x7fbfdc0fcef0 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfd9d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fe0e0 0x7fbfdc0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfd9d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fe0e0 0x7fbfdc0702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54848/0 (socket says 172.21.15.2:54848) 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.548+0000 7fbfd9d9b700 1 -- 172.21.15.2:0/3545934241 learned_addr learned my addr 172.21.15.2:0/3545934241 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfda59c700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfdc0fcef0 0x7fbfdc071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfd9d9b700 1 -- 172.21.15.2:0/3545934241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfdc0fcef0 msgr2=0x7fbfdc071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfd9d9b700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfdc0fcef0 0x7fbfdc071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfd9d9b700 1 -- 172.21.15.2:0/3545934241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc005040 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfd9d9b700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fe0e0 0x7fbfdc0702e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0002a50 tx=0x7fbfd00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfcaffd700 1 -- 172.21.15.2:0/3545934241 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd0007cc0 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc070d30 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfcaffd700 1 -- 172.21.15.2:0/3545934241 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfd000f040 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfcaffd700 1 -- 172.21.15.2:0/3545934241 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd000da30 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.549+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc1159a0 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.550+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc04bed0 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.551+0000 7fbfcaffd700 1 -- 172.21.15.2:0/3545934241 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fbfd000dbb0 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.551+0000 7fbfcaffd700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fbfc40692a0 0x7fbfc406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:49.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.551+0000 7fbfcaffd700 1 -- 172.21.15.2:0/3545934241 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbfd0086f90 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.551+0000 7fbfda59c700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fbfc40692a0 0x7fbfc406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:49.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.552+0000 7fbfda59c700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fbfc40692a0 0x7fbfc406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc00b840 tx=0x7fbfcc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:49.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.556+0000 7fbfcaffd700 1 -- 172.21.15.2:0/3545934241 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbfd0056c80 con 0x7fbfdc0fe0e0 2024-02-16T22:19:49.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.776+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfdc102ac0 con 0x7fbfc40692a0 2024-02-16T22:19:49.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.777+0000 7fbfcaffd700 1 -- 172.21.15.2:0/3545934241 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fbfdc102ac0 con 0x7fbfc40692a0 2024-02-16T22:19:49.778 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-16T22:19:49.778 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", 2024-02-16T22:19:49.779 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-16T22:19:49.779 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-16T22:19:49.779 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-02-16T22:19:49.779 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "", 2024-02-16T22:19:49.779 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 image", 2024-02-16T22:19:49.779 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-16T22:19:49.779 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-16T22:19:49.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fbfc40692a0 msgr2=0x7fbfc406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fbfc40692a0 0x7fbfc406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc00b840 tx=0x7fbfcc009e30 comp rx=0 tx=0).stop 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fe0e0 msgr2=0x7fbfdc0702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fe0e0 0x7fbfdc0702e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0002a50 tx=0x7fbfd00046b0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 shutdown_connections 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fbfc40692a0 0x7fbfc406b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfdc0fcef0 0x7fbfdc071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 --2- 172.21.15.2:0/3545934241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbfdc0fe0e0 0x7fbfdc0702e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 >> 172.21.15.2:0/3545934241 conn(0x7fbfdc0f8620 msgr2=0x7fbfdc1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 shutdown_connections 2024-02-16T22:19:49.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.779+0000 7fbfe0899700 1 -- 172.21.15.2:0/3545934241 wait complete. 2024-02-16T22:19:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.885+0000 7fa4d909c700 1 -- 172.21.15.2:0/1428588694 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 msgr2=0x7fa4d40fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.885+0000 7fa4d909c700 1 --2- 172.21.15.2:0/1428588694 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 0x7fa4d40fc250 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4009ce0 tx=0x7fa4c4004d10 comp rx=0 tx=0).stop 2024-02-16T22:19:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.885+0000 7fa4d909c700 1 -- 172.21.15.2:0/1428588694 shutdown_connections 2024-02-16T22:19:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.885+0000 7fa4d909c700 1 --2- 172.21.15.2:0/1428588694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d40fcfb0 0x7fa4d40fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.885+0000 7fa4d909c700 1 --2- 172.21.15.2:0/1428588694 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 0x7fa4d40fc250 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.885+0000 7fa4d909c700 1 -- 172.21.15.2:0/1428588694 >> 172.21.15.2:0/1428588694 conn(0x7fa4d40f7520 msgr2=0x7fa4d40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.885+0000 7fa4d909c700 1 -- 172.21.15.2:0/1428588694 shutdown_connections 2024-02-16T22:19:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.885+0000 7fa4d909c700 1 -- 172.21.15.2:0/1428588694 wait complete. 2024-02-16T22:19:49.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d909c700 1 Processor -- start 2024-02-16T22:19:49.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d909c700 1 -- start start 2024-02-16T22:19:49.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d909c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 0x7fa4d4110650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:49.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d909c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d40fcfb0 0x7fa4d4112ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:49.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d909c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d4066cf0 con 0x7fa4d40fbe40 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d909c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d4066e30 con 0x7fa4d40fcfb0 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d2d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 0x7fa4d4110650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d2d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 0x7fa4d4110650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54860/0 (socket says 172.21.15.2:54860) 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d2d9d700 1 -- 172.21.15.2:0/1794388424 learned_addr learned my addr 172.21.15.2:0/1794388424 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d259c700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d40fcfb0 0x7fa4d4112ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d2d9d700 1 -- 172.21.15.2:0/1794388424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d40fcfb0 msgr2=0x7fa4d4112ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d2d9d700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d40fcfb0 0x7fa4d4112ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d2d9d700 1 -- 172.21.15.2:0/1794388424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c4009940 con 0x7fa4d40fbe40 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d259c700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d40fcfb0 0x7fa4d4112ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.886+0000 7fa4d2d9d700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 0x7fa4d4110650 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4009380 tx=0x7fa4c4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.887+0000 7fa4cb7fe700 1 -- 172.21.15.2:0/1794388424 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c400e040 con 0x7fa4d40fbe40 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.887+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d41131a0 con 0x7fa4d40fbe40 2024-02-16T22:19:49.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.887+0000 7fa4cb7fe700 1 -- 172.21.15.2:0/1794388424 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4c401cd00 con 0x7fa4d40fbe40 2024-02-16T22:19:49.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.887+0000 7fa4cb7fe700 1 -- 172.21.15.2:0/1794388424 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c4012c30 con 0x7fa4d40fbe40 2024-02-16T22:19:49.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.887+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d41136e0 con 0x7fa4d40fbe40 2024-02-16T22:19:49.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.888+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d404bed0 con 0x7fa4d40fbe40 2024-02-16T22:19:49.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.888+0000 7fa4cb7fe700 1 -- 172.21.15.2:0/1794388424 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa4c4012410 con 0x7fa4d40fbe40 2024-02-16T22:19:49.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.888+0000 7fa4cb7fe700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa4c00692a0 0x7fa4c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:19:49.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.889+0000 7fa4cb7fe700 1 -- 172.21.15.2:0/1794388424 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa4c40895f0 con 0x7fa4d40fbe40 2024-02-16T22:19:49.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.889+0000 7fa4d259c700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa4c00692a0 0x7fa4c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:19:49.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.889+0000 7fa4d259c700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa4c00692a0 0x7fa4c006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc004600 tx=0x7fa4bc00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:19:49.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:49.893+0000 7fa4cb7fe700 1 -- 172.21.15.2:0/1794388424 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa4c40558d0 con 0x7fa4d40fbe40 2024-02-16T22:19:50.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.177+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa4d4104930 con 0x7fa4d40fbe40 2024-02-16T22:19:50.179 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:49 smithi002 ceph-mon[29154]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:50.179 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:49 smithi002 ceph-mon[29154]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:50.179 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:49 smithi002 ceph-mon[29154]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:19:50.179 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:49 smithi002 ceph-mon[29154]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:50.179 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:49 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1129552929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-16T22:19:50.179 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:49 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3664765214' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:19:50.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.177+0000 7fa4cb7fe700 1 -- 172.21.15.2:0/1794388424 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fa4c4058ef0 con 0x7fa4d40fbe40 2024-02-16T22:19:50.180 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-16T22:19:50.180 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-16T22:19:50.180 INFO:teuthology.orchestra.run.smithi002.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-16T22:19:50.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa4c00692a0 msgr2=0x7fa4c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:50.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa4c00692a0 0x7fa4c006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc004600 tx=0x7fa4bc00b040 comp rx=0 tx=0).stop 2024-02-16T22:19:50.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 msgr2=0x7fa4d4110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:19:50.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 0x7fa4d4110650 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4009380 tx=0x7fa4c4004d10 comp rx=0 tx=0).stop 2024-02-16T22:19:50.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 shutdown_connections 2024-02-16T22:19:50.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa4c00692a0 0x7fa4c006b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:50.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4d40fbe40 0x7fa4d4110650 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:50.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 --2- 172.21.15.2:0/1794388424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa4d40fcfb0 0x7fa4d4112ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:19:50.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 >> 172.21.15.2:0/1794388424 conn(0x7fa4d40f7520 msgr2=0x7fa4d4100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:19:50.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 shutdown_connections 2024-02-16T22:19:50.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:19:50.179+0000 7fa4d909c700 1 -- 172.21.15.2:0/1794388424 wait complete. 2024-02-16T22:19:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:49 smithi103 ceph-mon[36259]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:49 smithi103 ceph-mon[36259]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:49 smithi103 ceph-mon[36259]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:19:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:49 smithi103 ceph-mon[36259]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:49 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1129552929' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-16T22:19:50.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:49 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3664765214' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:19:51.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:50 smithi002 ceph-mon[29154]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:51.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:50 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1794388424' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-16T22:19:51.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:50 smithi103 ceph-mon[36259]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:19:51.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:50 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1794388424' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-16T22:19:52.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:51 smithi002 ceph-mon[29154]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:19:52.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:51 smithi103 ceph-mon[36259]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:19:54.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:53 smithi002 ceph-mon[29154]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:19:54.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:19:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:53 smithi103 ceph-mon[36259]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:19:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:19:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:55 smithi002 ceph-mon[29154]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:19:56.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:55 smithi103 ceph-mon[36259]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:19:56.994 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:56 smithi002 ceph-mon[29154]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:19:57.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:56 smithi103 ceph-mon[36259]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:19:59.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:19:59 smithi002 ceph-mon[29154]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:00.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:19:59 smithi103 ceph-mon[36259]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:01.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:00 smithi002 ceph-mon[29154]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-16T22:20:01.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:00 smithi002 ceph-mon[29154]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-16T22:20:01.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:00 smithi002 ceph-mon[29154]: fs cephfs has deprecated feature inline_data enabled. 2024-02-16T22:20:01.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:00 smithi103 ceph-mon[36259]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-16T22:20:01.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:00 smithi103 ceph-mon[36259]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-16T22:20:01.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:00 smithi103 ceph-mon[36259]: fs cephfs has deprecated feature inline_data enabled. 2024-02-16T22:20:02.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:01 smithi002 ceph-mon[29154]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:02.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:01 smithi103 ceph-mon[36259]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:03.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:02 smithi002 ceph-mon[29154]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:03.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:02 smithi103 ceph-mon[36259]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:06.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:05 smithi002 ceph-mon[29154]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-16T22:20:06.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:05 smithi103 ceph-mon[36259]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-16T22:20:08.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:07 smithi002 ceph-mon[29154]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2024-02-16T22:20:08.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:07 smithi103 ceph-mon[36259]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2024-02-16T22:20:09.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:20:09.226 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:08 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:20:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:09 smithi002 ceph-mon[29154]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2024-02-16T22:20:10.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:09 smithi103 ceph-mon[36259]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2024-02-16T22:20:11.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:10 smithi002 ceph-mon[29154]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-16T22:20:11.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:10 smithi103 ceph-mon[36259]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-16T22:20:13.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:13 smithi002 ceph-mon[29154]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-16T22:20:14.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:13 smithi103 ceph-mon[36259]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-16T22:20:16.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:15 smithi002 ceph-mon[29154]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-16T22:20:16.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:15 smithi103 ceph-mon[36259]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-16T22:20:18.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:17 smithi002 ceph-mon[29154]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2024-02-16T22:20:18.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:17 smithi103 ceph-mon[36259]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2024-02-16T22:20:20.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:19 smithi002 ceph-mon[29154]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2024-02-16T22:20:20.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 -- 172.21.15.2:0/4109540824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 msgr2=0x7f29c00fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 --2- 172.21.15.2:0/4109540824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 0x7f29c00fb630 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f29b0004940 tx=0x7f29b0009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 -- 172.21.15.2:0/4109540824 shutdown_connections 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 --2- 172.21.15.2:0/4109540824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29c00fe2b0 0x7f29c00fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 --2- 172.21.15.2:0/4109540824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 0x7f29c00fb630 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 -- 172.21.15.2:0/4109540824 >> 172.21.15.2:0/4109540824 conn(0x7f29c00f6ca0 msgr2=0x7f29c00f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 -- 172.21.15.2:0/4109540824 shutdown_connections 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 -- 172.21.15.2:0/4109540824 wait complete. 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 Processor -- start 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.287+0000 7f29c6846700 1 -- start start 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29c6846700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 0x7f29c010d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29c6846700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29c00fe2b0 0x7f29c010dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29c6846700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c010e4e0 con 0x7f29c00fb220 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29c6846700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c010ed80 con 0x7f29c00fe2b0 2024-02-16T22:20:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 0x7f29c010d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 0x7f29c010d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36484/0 (socket says 172.21.15.2:36484) 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bffff700 1 -- 172.21.15.2:0/2272214885 learned_addr learned my addr 172.21.15.2:0/2272214885 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bf7fe700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29c00fe2b0 0x7f29c010dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bffff700 1 -- 172.21.15.2:0/2272214885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29c00fe2b0 msgr2=0x7f29c010dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bffff700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29c00fe2b0 0x7f29c010dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bffff700 1 -- 172.21.15.2:0/2272214885 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0005040 con 0x7f29c00fb220 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bffff700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 0x7f29c010d9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009fd0 tx=0x7f29b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bcff9700 1 -- 172.21.15.2:0/2272214885 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29b0021070 con 0x7f29c00fb220 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c010f000 con 0x7f29c00fb220 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bcff9700 1 -- 172.21.15.2:0/2272214885 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29b001c400 con 0x7f29c00fb220 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.288+0000 7f29bcff9700 1 -- 172.21.15.2:0/2272214885 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29b000e070 con 0x7f29c00fb220 2024-02-16T22:20:20.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.289+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c010f4e0 con 0x7f29c00fb220 2024-02-16T22:20:20.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.289+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c004bed0 con 0x7f29c00fb220 2024-02-16T22:20:20.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.290+0000 7f29bcff9700 1 -- 172.21.15.2:0/2272214885 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f29b001ca90 con 0x7f29c00fb220 2024-02-16T22:20:20.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.290+0000 7f29bcff9700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29a80692a0 0x7f29a806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:20.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.290+0000 7f29bcff9700 1 -- 172.21.15.2:0/2272214885 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f29b0088310 con 0x7f29c00fb220 2024-02-16T22:20:20.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.290+0000 7f29bf7fe700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29a80692a0 0x7f29a806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:20.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.291+0000 7f29bf7fe700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29a80692a0 0x7f29a806b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009b80 tx=0x7f29b400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:20.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.295+0000 7f29bcff9700 1 -- 172.21.15.2:0/2272214885 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f29b0058080 con 0x7f29c00fb220 2024-02-16T22:20:20.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:19 smithi103 ceph-mon[36259]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2024-02-16T22:20:20.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.531+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f29c0102c00 con 0x7f29a80692a0 2024-02-16T22:20:20.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.532+0000 7f29bcff9700 1 -- 172.21.15.2:0/2272214885 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f29c0102c00 con 0x7f29a80692a0 2024-02-16T22:20:20.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.534+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29a80692a0 msgr2=0x7f29a806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.534+0000 7f29c6846700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29a80692a0 0x7f29a806b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009b80 tx=0x7f29b400b040 comp rx=0 tx=0).stop 2024-02-16T22:20:20.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.534+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 msgr2=0x7f29c010d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.534+0000 7f29c6846700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 0x7f29c010d9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009fd0 tx=0x7f29b0009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:20.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.534+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 shutdown_connections 2024-02-16T22:20:20.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.534+0000 7f29c6846700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29a80692a0 0x7f29a806b750 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.534+0000 7f29c6846700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c00fb220 0x7f29c010d9a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.534+0000 7f29c6846700 1 --2- 172.21.15.2:0/2272214885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f29c00fe2b0 0x7f29c010dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.534+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 >> 172.21.15.2:0/2272214885 conn(0x7f29c00f6ca0 msgr2=0x7f29c01014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:20.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.535+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 shutdown_connections 2024-02-16T22:20:20.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.535+0000 7f29c6846700 1 -- 172.21.15.2:0/2272214885 wait complete. 2024-02-16T22:20:20.546 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-16T22:20:20.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.642+0000 7fe97fb39700 1 -- 172.21.15.2:0/916274537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 msgr2=0x7fe9780fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.642+0000 7fe97fb39700 1 --2- 172.21.15.2:0/916274537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 0x7fe9780fc250 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe968004940 tx=0x7fe968009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:20.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.642+0000 7fe97fb39700 1 -- 172.21.15.2:0/916274537 shutdown_connections 2024-02-16T22:20:20.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.642+0000 7fe97fb39700 1 --2- 172.21.15.2:0/916274537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9780fcfb0 0x7fe9780fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.642+0000 7fe97fb39700 1 --2- 172.21.15.2:0/916274537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 0x7fe9780fc250 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.642+0000 7fe97fb39700 1 -- 172.21.15.2:0/916274537 >> 172.21.15.2:0/916274537 conn(0x7fe9780f74e0 msgr2=0x7fe9780f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:20.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.642+0000 7fe97fb39700 1 -- 172.21.15.2:0/916274537 shutdown_connections 2024-02-16T22:20:20.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.642+0000 7fe97fb39700 1 -- 172.21.15.2:0/916274537 wait complete. 2024-02-16T22:20:20.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.642+0000 7fe97fb39700 1 Processor -- start 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97fb39700 1 -- start start 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97fb39700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 0x7fe9781105f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97fb39700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9780fcfb0 0x7fe978112b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97fb39700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe978066460 con 0x7fe9780fbe40 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97fb39700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780665a0 con 0x7fe9780fcfb0 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97d8d5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 0x7fe9781105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97d8d5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 0x7fe9781105f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39234/0 (socket says 172.21.15.2:39234) 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97d8d5700 1 -- 172.21.15.2:0/4281436634 learned_addr learned my addr 172.21.15.2:0/4281436634 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97d0d4700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9780fcfb0 0x7fe978112b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97d8d5700 1 -- 172.21.15.2:0/4281436634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9780fcfb0 msgr2=0x7fe978112b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97d8d5700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9780fcfb0 0x7fe978112b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97d8d5700 1 -- 172.21.15.2:0/4281436634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968005040 con 0x7fe9780fbe40 2024-02-16T22:20:20.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe97d8d5700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 0x7fe9781105f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe9680093b0 tx=0x7fe968009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:20.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.643+0000 7fe96e7fc700 1 -- 172.21.15.2:0/4281436634 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe96800e040 con 0x7fe9780fbe40 2024-02-16T22:20:20.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.644+0000 7fe96e7fc700 1 -- 172.21.15.2:0/4281436634 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe96800bcc0 con 0x7fe9780fbe40 2024-02-16T22:20:20.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.644+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe978113140 con 0x7fe9780fbe40 2024-02-16T22:20:20.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.644+0000 7fe96e7fc700 1 -- 172.21.15.2:0/4281436634 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe96801c630 con 0x7fe9780fbe40 2024-02-16T22:20:20.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.644+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe978113680 con 0x7fe9780fbe40 2024-02-16T22:20:20.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.645+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97804bed0 con 0x7fe9780fbe40 2024-02-16T22:20:20.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.645+0000 7fe96e7fc700 1 -- 172.21.15.2:0/4281436634 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe968013980 con 0x7fe9780fbe40 2024-02-16T22:20:20.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.646+0000 7fe96e7fc700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe9640692a0 0x7fe96406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:20.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.646+0000 7fe96e7fc700 1 -- 172.21.15.2:0/4281436634 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe968088560 con 0x7fe9780fbe40 2024-02-16T22:20:20.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.646+0000 7fe97d0d4700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe9640692a0 0x7fe96406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:20.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.646+0000 7fe97d0d4700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe9640692a0 0x7fe96406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe974009fd0 tx=0x7fe97400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:20.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.650+0000 7fe96e7fc700 1 -- 172.21.15.2:0/4281436634 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe968054840 con 0x7fe9780fbe40 2024-02-16T22:20:20.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.868+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe978101990 con 0x7fe9640692a0 2024-02-16T22:20:20.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.869+0000 7fe96e7fc700 1 -- 172.21.15.2:0/4281436634 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fe978101990 con 0x7fe9640692a0 2024-02-16T22:20:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.871+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe9640692a0 msgr2=0x7fe96406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.871+0000 7fe97fb39700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe9640692a0 0x7fe96406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe974009fd0 tx=0x7fe97400b040 comp rx=0 tx=0).stop 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.871+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 msgr2=0x7fe9781105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.871+0000 7fe97fb39700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 0x7fe9781105f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe9680093b0 tx=0x7fe968009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.871+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 shutdown_connections 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.871+0000 7fe97fb39700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe9640692a0 0x7fe96406b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.872+0000 7fe97fb39700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9780fbe40 0x7fe9781105f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.872+0000 7fe97fb39700 1 --2- 172.21.15.2:0/4281436634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9780fcfb0 0x7fe978112b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.872+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 >> 172.21.15.2:0/4281436634 conn(0x7fe9780f74e0 msgr2=0x7fe978100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.872+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 shutdown_connections 2024-02-16T22:20:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.872+0000 7fe97fb39700 1 -- 172.21.15.2:0/4281436634 wait complete. 2024-02-16T22:20:20.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.977+0000 7f66e4c3b700 1 -- 172.21.15.2:0/4239218229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 msgr2=0x7f66e0104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.977+0000 7f66e4c3b700 1 --2- 172.21.15.2:0/4239218229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 0x7f66e0104810 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f66d4004940 tx=0x7f66d4009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:20.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 -- 172.21.15.2:0/4239218229 shutdown_connections 2024-02-16T22:20:20.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 --2- 172.21.15.2:0/4239218229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 0x7f66e0104810 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 --2- 172.21.15.2:0/4239218229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0066c70 0x7f66e0101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 -- 172.21.15.2:0/4239218229 >> 172.21.15.2:0/4239218229 conn(0x7f66e00f7520 msgr2=0x7f66e00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:20.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 -- 172.21.15.2:0/4239218229 shutdown_connections 2024-02-16T22:20:20.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 -- 172.21.15.2:0/4239218229 wait complete. 2024-02-16T22:20:20.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 Processor -- start 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 -- start start 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0066c70 0x7f66e010fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 0x7f66e0110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e0110620 con 0x7f66e0102430 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66e4c3b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e0110ec0 con 0x7f66e0066c70 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66ddd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 0x7f66e0110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66ddd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 0x7f66e0110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39240/0 (socket says 172.21.15.2:39240) 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66ddd9b700 1 -- 172.21.15.2:0/554977846 learned_addr learned my addr 172.21.15.2:0/554977846 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.978+0000 7f66de59c700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0066c70 0x7f66e010fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.979+0000 7f66ddd9b700 1 -- 172.21.15.2:0/554977846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0066c70 msgr2=0x7f66e010fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.979+0000 7f66ddd9b700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0066c70 0x7f66e010fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.979+0000 7f66ddd9b700 1 -- 172.21.15.2:0/554977846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d4005040 con 0x7f66e0102430 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.979+0000 7f66ddd9b700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 0x7f66e0110020 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f66d4000c00 tx=0x7f66d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.979+0000 7f66ceffd700 1 -- 172.21.15.2:0/554977846 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66d400e040 con 0x7f66e0102430 2024-02-16T22:20:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.979+0000 7f66ceffd700 1 -- 172.21.15.2:0/554977846 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66d400bbf0 con 0x7f66e0102430 2024-02-16T22:20:20.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.979+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e0111140 con 0x7f66e0102430 2024-02-16T22:20:20.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.979+0000 7f66ceffd700 1 -- 172.21.15.2:0/554977846 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66d401c630 con 0x7f66e0102430 2024-02-16T22:20:20.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.979+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e0111680 con 0x7f66e0102430 2024-02-16T22:20:20.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.980+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e0109c50 con 0x7f66e0102430 2024-02-16T22:20:20.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.980+0000 7f66ceffd700 1 -- 172.21.15.2:0/554977846 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f66d400bd60 con 0x7f66e0102430 2024-02-16T22:20:20.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.981+0000 7f66ceffd700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f66c80692a0 0x7f66c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:20.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.981+0000 7f66ceffd700 1 -- 172.21.15.2:0/554977846 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f66d40884c0 con 0x7f66e0102430 2024-02-16T22:20:20.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.981+0000 7f66de59c700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f66c80692a0 0x7f66c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:20.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.981+0000 7f66de59c700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f66c80692a0 0x7f66c806b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009fd0 tx=0x7f66d000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:20.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:20.985+0000 7f66ceffd700 1 -- 172.21.15.2:0/554977846 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f66d4058250 con 0x7f66e0102430 2024-02-16T22:20:21.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.192+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f66e01118f0 con 0x7f66c80692a0 2024-02-16T22:20:21.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.204+0000 7f66ceffd700 1 -- 172.21.15.2:0/554977846 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f66e01118f0 con 0x7f66c80692a0 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.smithi002 smithi002 *:9093,9094 running (5m) 58s ago 6m 25.0M - 0.25.0 c8568f914cd2 2b7780bcc981 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi002 smithi002 running (7m) 58s ago 7m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 c1d597f23cdd 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi103 smithi103 running (6m) 63s ago 6m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 db2a59e09a51 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.smithi002 smithi002 *:3000 running (5m) 58s ago 6m 87.3M - 9.4.7 954c08fa6188 14099b699363 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi002.hhufhl smithi002 running (77s) 58s ago 76s 15.3M - 17.2.7-513-gb88fae66 870a0944ccb8 12d8ab360b57 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi002.llkqse smithi002 running (85s) 58s ago 84s 10.5M - 17.2.7-513-gb88fae66 870a0944ccb8 8e266571411c 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi103.epsgnr smithi103 running (74s) 63s ago 74s 14.5M - 17.2.7-513-gb88fae66 870a0944ccb8 d43d0b5d182c 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi103.kbizjo smithi103 running (82s) 63s ago 82s 14.6M - 17.2.7-513-gb88fae66 870a0944ccb8 86d4a2e35372 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi002.resmjv smithi002 *:9283 running (8m) 58s ago 8m 480M - 17.2.7-513-gb88fae66 870a0944ccb8 f9b132a0aa69 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi103.ydydbo smithi103 *:8443,9283 running (5m) 63s ago 5m 433M - 17.2.7-513-gb88fae66 870a0944ccb8 463897d4efc5 2024-02-16T22:20:21.206 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi002 smithi002 running (8m) 58s ago 8m 45.2M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 cc6b0b4bb587 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi103 smithi103 running (5m) 63s ago 5m 35.6M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 3d198bee2a92 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi002 smithi002 *:9100 running (6m) 58s ago 6m 15.9M - 1.5.0 0da6a335fe13 e6146a57c445 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi103 smithi103 *:9100 running (5m) 63s ago 5m 17.0M - 1.5.0 0da6a335fe13 0b21b215b785 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (5m) 58s ago 5m 48.7M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 a3f19d068baf 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (4m) 58s ago 4m 46.1M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 58697cab97b0 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (4m) 58s ago 4m 47.1M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 634798c2dbad 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi103 running (3m) 63s ago 3m 43.9M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 a5332c698672 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi103 running (3m) 63s ago 3m 42.5M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 fa2a12f4b710 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi103 running (2m) 63s ago 2m 42.1M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 175f4eefd1f6 2024-02-16T22:20:21.207 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.smithi002 smithi002 *:9095 running (5m) 58s ago 6m 55.1M - 2.43.0 a07b618ecd1d d03f44725127 2024-02-16T22:20:21.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f66c80692a0 msgr2=0x7f66c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f66c80692a0 0x7f66c806b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009fd0 tx=0x7f66d000b040 comp rx=0 tx=0).stop 2024-02-16T22:20:21.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 msgr2=0x7f66e0110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 0x7f66e0110020 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f66d4000c00 tx=0x7f66d4009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:21.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 shutdown_connections 2024-02-16T22:20:21.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f66c80692a0 0x7f66c806b750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66e0066c70 0x7f66e010fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 --2- 172.21.15.2:0/554977846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66e0102430 0x7f66e0110020 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 >> 172.21.15.2:0/554977846 conn(0x7f66e00f7520 msgr2=0x7f66e00f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:21.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 shutdown_connections 2024-02-16T22:20:21.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.206+0000 7f66e4c3b700 1 -- 172.21.15.2:0/554977846 wait complete. 2024-02-16T22:20:21.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.309+0000 7fe609ae7700 1 -- 172.21.15.2:0/3837625366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 msgr2=0x7fe6040fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.309+0000 7fe609ae7700 1 --2- 172.21.15.2:0/3837625366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 0x7fe6040fd320 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4004940 tx=0x7fe5f4009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:21.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.309+0000 7fe609ae7700 1 -- 172.21.15.2:0/3837625366 shutdown_connections 2024-02-16T22:20:21.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.309+0000 7fe609ae7700 1 --2- 172.21.15.2:0/3837625366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6040fe0e0 0x7fe6040fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.309+0000 7fe609ae7700 1 --2- 172.21.15.2:0/3837625366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 0x7fe6040fd320 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.309+0000 7fe609ae7700 1 -- 172.21.15.2:0/3837625366 >> 172.21.15.2:0/3837625366 conn(0x7fe6040f8620 msgr2=0x7fe6040faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:21.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.309+0000 7fe609ae7700 1 -- 172.21.15.2:0/3837625366 shutdown_connections 2024-02-16T22:20:21.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.309+0000 7fe609ae7700 1 -- 172.21.15.2:0/3837625366 wait complete. 2024-02-16T22:20:21.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe609ae7700 1 Processor -- start 2024-02-16T22:20:21.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe609ae7700 1 -- start start 2024-02-16T22:20:21.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe609ae7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 0x7fe604071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe609ae7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6040fe0e0 0x7fe6040702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe609ae7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe604070970 con 0x7fe6040fcef0 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe609ae7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe604070ae0 con 0x7fe6040fe0e0 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe6037fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 0x7fe604071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe6037fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 0x7fe604071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39252/0 (socket says 172.21.15.2:39252) 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe6037fe700 1 -- 172.21.15.2:0/3975046050 learned_addr learned my addr 172.21.15.2:0/3975046050 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.310+0000 7fe602ffd700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6040fe0e0 0x7fe6040702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.311+0000 7fe6037fe700 1 -- 172.21.15.2:0/3975046050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6040fe0e0 msgr2=0x7fe6040702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.311+0000 7fe6037fe700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6040fe0e0 0x7fe6040702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.311+0000 7fe6037fe700 1 -- 172.21.15.2:0/3975046050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f4005040 con 0x7fe6040fcef0 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.311+0000 7fe6037fe700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 0x7fe604071e60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400b3b0 tx=0x7fe5f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.311+0000 7fe5ebfff700 1 -- 172.21.15.2:0/3975046050 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5f400e040 con 0x7fe6040fcef0 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.311+0000 7fe5ebfff700 1 -- 172.21.15.2:0/3975046050 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5f400bcc0 con 0x7fe6040fcef0 2024-02-16T22:20:21.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.311+0000 7fe5ebfff700 1 -- 172.21.15.2:0/3975046050 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5f401c630 con 0x7fe6040fcef0 2024-02-16T22:20:21.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.311+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe604070d60 con 0x7fe6040fcef0 2024-02-16T22:20:21.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.311+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6041159e0 con 0x7fe6040fcef0 2024-02-16T22:20:21.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.312+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60404bed0 con 0x7fe6040fcef0 2024-02-16T22:20:21.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.313+0000 7fe5ebfff700 1 -- 172.21.15.2:0/3975046050 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe5f4013980 con 0x7fe6040fcef0 2024-02-16T22:20:21.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.313+0000 7fe5ebfff700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5ec0692a0 0x7fe5ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:21.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.313+0000 7fe602ffd700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5ec0692a0 0x7fe5ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:21.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.313+0000 7fe5ebfff700 1 -- 172.21.15.2:0/3975046050 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe5f40891b0 con 0x7fe6040fcef0 2024-02-16T22:20:21.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.313+0000 7fe602ffd700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5ec0692a0 0x7fe5ec06b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009bf0 tx=0x7fe5f80091a0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:21.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.317+0000 7fe5ebfff700 1 -- 172.21.15.2:0/3975046050 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe5f4058f40 con 0x7fe6040fcef0 2024-02-16T22:20:21.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.609+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe604115bd0 con 0x7fe6040fcef0 2024-02-16T22:20:21.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.609+0000 7fe5ebfff700 1 -- 172.21.15.2:0/3975046050 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fe604115bd0 con 0x7fe6040fcef0 2024-02-16T22:20:21.611 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-16T22:20:21.611 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-16T22:20:21.611 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 2 2024-02-16T22:20:21.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:20:21.611 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-16T22:20:21.611 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 2 2024-02-16T22:20:21.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 6 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": { 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 4 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 14 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-16T22:20:21.612 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5ec0692a0 msgr2=0x7fe5ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5ec0692a0 0x7fe5ec06b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009bf0 tx=0x7fe5f80091a0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 msgr2=0x7fe604071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 0x7fe604071e60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400b3b0 tx=0x7fe5f4009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 shutdown_connections 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe5ec0692a0 0x7fe5ec06b750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6040fcef0 0x7fe604071e60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 --2- 172.21.15.2:0/3975046050 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe6040fe0e0 0x7fe6040702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 >> 172.21.15.2:0/3975046050 conn(0x7fe6040f8620 msgr2=0x7fe6041013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:21.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 shutdown_connections 2024-02-16T22:20:21.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.611+0000 7fe609ae7700 1 -- 172.21.15.2:0/3975046050 wait complete. 2024-02-16T22:20:21.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.716+0000 7f3838322700 1 -- 172.21.15.2:0/4056095454 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 msgr2=0x7f38300fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.716+0000 7f3838322700 1 --2- 172.21.15.2:0/4056095454 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 0x7f38300fd420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3824004940 tx=0x7f3824009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:21.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.716+0000 7f3838322700 1 -- 172.21.15.2:0/4056095454 shutdown_connections 2024-02-16T22:20:21.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.716+0000 7f3838322700 1 --2- 172.21.15.2:0/4056095454 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 0x7f38300fd420 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.716+0000 7f3838322700 1 --2- 172.21.15.2:0/4056095454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38300fbe40 0x7f38300fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.716+0000 7f3838322700 1 -- 172.21.15.2:0/4056095454 >> 172.21.15.2:0/4056095454 conn(0x7f38300f74e0 msgr2=0x7f38300f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:21.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.716+0000 7f3838322700 1 -- 172.21.15.2:0/4056095454 shutdown_connections 2024-02-16T22:20:21.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.716+0000 7f3838322700 1 -- 172.21.15.2:0/4056095454 wait complete. 2024-02-16T22:20:21.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f3838322700 1 Processor -- start 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f3838322700 1 -- start start 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f3838322700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38300fbe40 0x7f383010d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f3838322700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 0x7f383010ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f3838322700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383010e3f0 con 0x7f38300fcfb0 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f3838322700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383010ec90 con 0x7f38300fbe40 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f38358bd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 0x7f383010ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f38358bd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 0x7f383010ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39268/0 (socket says 172.21.15.2:39268) 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f38358bd700 1 -- 172.21.15.2:0/2853226226 learned_addr learned my addr 172.21.15.2:0/2853226226 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f38360be700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38300fbe40 0x7f383010d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f38358bd700 1 -- 172.21.15.2:0/2853226226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38300fbe40 msgr2=0x7f383010d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f38358bd700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38300fbe40 0x7f383010d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.717+0000 7f38358bd700 1 -- 172.21.15.2:0/2853226226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3824005040 con 0x7f38300fcfb0 2024-02-16T22:20:21.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.718+0000 7f38358bd700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 0x7f383010ddf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3824000c00 tx=0x7f3824009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:21.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.718+0000 7f382effd700 1 -- 172.21.15.2:0/2853226226 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3824021070 con 0x7f38300fcfb0 2024-02-16T22:20:21.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.718+0000 7f382effd700 1 -- 172.21.15.2:0/2853226226 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f382401c400 con 0x7f38300fcfb0 2024-02-16T22:20:21.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.718+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383010ef10 con 0x7f38300fcfb0 2024-02-16T22:20:21.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.718+0000 7f382effd700 1 -- 172.21.15.2:0/2853226226 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f382400e070 con 0x7f38300fcfb0 2024-02-16T22:20:21.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.718+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383010f3f0 con 0x7f38300fcfb0 2024-02-16T22:20:21.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.719+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383004bed0 con 0x7f38300fcfb0 2024-02-16T22:20:21.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.719+0000 7f382effd700 1 -- 172.21.15.2:0/2853226226 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f382401ca50 con 0x7f38300fcfb0 2024-02-16T22:20:21.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.720+0000 7f382effd700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f38180692a0 0x7f381806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:21.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.720+0000 7f382effd700 1 -- 172.21.15.2:0/2853226226 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f38240882e0 con 0x7f38300fcfb0 2024-02-16T22:20:21.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.720+0000 7f38360be700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f38180692a0 0x7f381806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:21.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.720+0000 7f38360be700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f38180692a0 0x7f381806b750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3820001470 tx=0x7f382000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:21.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.724+0000 7f382effd700 1 -- 172.21.15.2:0/2853226226 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f38240580f0 con 0x7f38300fcfb0 2024-02-16T22:20:21.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.974+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f38300659e0 con 0x7f38300fcfb0 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.974+0000 7f382effd700 1 -- 172.21.15.2:0/2853226226 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1830 (secure 0 0 0) 0x7f3824057c80 con 0x7f38300fcfb0 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:e16 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 16 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-16T22:20:21.976 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-02-16T22:18:52.996722+0000 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2024-02-16T22:19:10.960547+0000 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 60 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 46 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 1 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:in 0 2024-02-16T22:20:21.977 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24263} 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3] 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data enabled 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 1 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.kbizjo{0:24263} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.hhufhl{0:14486} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.2:6828/971745533,v1:172.21.15.2:6829/971745533] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2024-02-16T22:20:21.978 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:20:21.979 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.llkqse{-1:14496} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.2:6826/3769723051,v1:172.21.15.2:6827/3769723051] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:20:21.979 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.epsgnr{-1:24271} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.103:6826/1310923282,v1:172.21.15.103:6827/1310923282] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:20:21.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f38180692a0 msgr2=0x7f381806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f38180692a0 0x7f381806b750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3820001470 tx=0x7f382000b040 comp rx=0 tx=0).stop 2024-02-16T22:20:21.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 msgr2=0x7f383010ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 0x7f383010ddf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3824000c00 tx=0x7f3824009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 shutdown_connections 2024-02-16T22:20:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f38180692a0 0x7f381806b750 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38300fbe40 0x7f383010d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 --2- 172.21.15.2:0/2853226226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38300fcfb0 0x7f383010ddf0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 >> 172.21.15.2:0/2853226226 conn(0x7f38300f74e0 msgr2=0x7f38300f9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 shutdown_connections 2024-02-16T22:20:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:21.976+0000 7f3838322700 1 -- 172.21.15.2:0/2853226226 wait complete. 2024-02-16T22:20:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 16 2024-02-16T22:20:22.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.084+0000 7fcf27749700 1 -- 172.21.15.2:0/1356197220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 msgr2=0x7fcf200fd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:22.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.084+0000 7fcf27749700 1 --2- 172.21.15.2:0/1356197220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 0x7fcf200fd4f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcf10004940 tx=0x7fcf10009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:22.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 -- 172.21.15.2:0/1356197220 shutdown_connections 2024-02-16T22:20:22.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 --2- 172.21.15.2:0/1356197220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 0x7fcf200fd4f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 --2- 172.21.15.2:0/1356197220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf200fbe80 0x7fcf200fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 -- 172.21.15.2:0/1356197220 >> 172.21.15.2:0/1356197220 conn(0x7fcf200f7590 msgr2=0x7fcf200f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:22.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 -- 172.21.15.2:0/1356197220 shutdown_connections 2024-02-16T22:20:22.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 -- 172.21.15.2:0/1356197220 wait complete. 2024-02-16T22:20:22.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 Processor -- start 2024-02-16T22:20:22.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 -- start start 2024-02-16T22:20:22.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf200fbe80 0x7fcf201105f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:22.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 0x7fcf20112b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:22.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf20066d30 con 0x7fcf200fd080 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.085+0000 7fcf27749700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf20066e70 con 0x7fcf200fbe80 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf24ce4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 0x7fcf20112b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf24ce4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 0x7fcf20112b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39284/0 (socket says 172.21.15.2:39284) 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf24ce4700 1 -- 172.21.15.2:0/1861648684 learned_addr learned my addr 172.21.15.2:0/1861648684 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf254e5700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf200fbe80 0x7fcf201105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf24ce4700 1 -- 172.21.15.2:0/1861648684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf200fbe80 msgr2=0x7fcf201105f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf24ce4700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf200fbe80 0x7fcf201105f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf24ce4700 1 -- 172.21.15.2:0/1861648684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf10005040 con 0x7fcf200fd080 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf24ce4700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 0x7fcf20112b40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcf10000c00 tx=0x7fcf10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf15ffb700 1 -- 172.21.15.2:0/1861648684 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf10021070 con 0x7fcf200fd080 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf15ffb700 1 -- 172.21.15.2:0/1861648684 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf1001c400 con 0x7fcf200fd080 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf20113140 con 0x7fcf200fd080 2024-02-16T22:20:22.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.086+0000 7fcf15ffb700 1 -- 172.21.15.2:0/1861648684 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf1000e070 con 0x7fcf200fd080 2024-02-16T22:20:22.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.087+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf201135d0 con 0x7fcf200fd080 2024-02-16T22:20:22.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.087+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2004bed0 con 0x7fcf200fd080 2024-02-16T22:20:22.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.088+0000 7fcf15ffb700 1 -- 172.21.15.2:0/1861648684 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fcf1001ca50 con 0x7fcf200fd080 2024-02-16T22:20:22.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.088+0000 7fcf15ffb700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fcf0c0692a0 0x7fcf0c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:22.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.088+0000 7fcf15ffb700 1 -- 172.21.15.2:0/1861648684 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcf100883e0 con 0x7fcf200fd080 2024-02-16T22:20:22.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.088+0000 7fcf254e5700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fcf0c0692a0 0x7fcf0c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:22.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.089+0000 7fcf254e5700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fcf0c0692a0 0x7fcf0c06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcf200669b0 tx=0x7fcf1c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:22.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.093+0000 7fcf15ffb700 1 -- 172.21.15.2:0/1861648684 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcf100546c0 con 0x7fcf200fd080 2024-02-16T22:20:22.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:21 smithi002 ceph-mon[29154]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:22.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:21 smithi002 ceph-mon[29154]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 3 op/s 2024-02-16T22:20:22.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:21 smithi002 ceph-mon[29154]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:22.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:21 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3975046050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-16T22:20:22.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.318+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf201019d0 con 0x7fcf0c0692a0 2024-02-16T22:20:22.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.319+0000 7fcf15ffb700 1 -- 172.21.15.2:0/1861648684 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fcf201019d0 con 0x7fcf0c0692a0 2024-02-16T22:20:22.321 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-16T22:20:22.321 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", 2024-02-16T22:20:22.321 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-16T22:20:22.321 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-16T22:20:22.321 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-02-16T22:20:22.321 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "", 2024-02-16T22:20:22.321 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 image", 2024-02-16T22:20:22.321 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-16T22:20:22.321 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-16T22:20:22.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fcf0c0692a0 msgr2=0x7fcf0c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:22.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fcf0c0692a0 0x7fcf0c06b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcf200669b0 tx=0x7fcf1c00b040 comp rx=0 tx=0).stop 2024-02-16T22:20:22.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 msgr2=0x7fcf20112b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:22.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 0x7fcf20112b40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcf10000c00 tx=0x7fcf10009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:22.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 shutdown_connections 2024-02-16T22:20:22.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fcf0c0692a0 0x7fcf0c06b750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcf200fbe80 0x7fcf201105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 --2- 172.21.15.2:0/1861648684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf200fd080 0x7fcf20112b40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 >> 172.21.15.2:0/1861648684 conn(0x7fcf200f7590 msgr2=0x7fcf201002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:22.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 shutdown_connections 2024-02-16T22:20:22.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.321+0000 7fcf27749700 1 -- 172.21.15.2:0/1861648684 wait complete. 2024-02-16T22:20:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:21 smithi103 ceph-mon[36259]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:21 smithi103 ceph-mon[36259]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 3 op/s 2024-02-16T22:20:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:21 smithi103 ceph-mon[36259]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:22.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:21 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3975046050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-16T22:20:22.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.425+0000 7fa9b3332700 1 -- 172.21.15.2:0/1003700704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 msgr2=0x7fa9ac0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:22.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.425+0000 7fa9b3332700 1 --2- 172.21.15.2:0/1003700704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 0x7fa9ac0fd320 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009ce0 tx=0x7fa9a8004d10 comp rx=0 tx=0).stop 2024-02-16T22:20:22.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 -- 172.21.15.2:0/1003700704 shutdown_connections 2024-02-16T22:20:22.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 --2- 172.21.15.2:0/1003700704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac0fe0e0 0x7fa9ac0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 --2- 172.21.15.2:0/1003700704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 0x7fa9ac0fd320 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 -- 172.21.15.2:0/1003700704 >> 172.21.15.2:0/1003700704 conn(0x7fa9ac0f8620 msgr2=0x7fa9ac0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:22.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 -- 172.21.15.2:0/1003700704 shutdown_connections 2024-02-16T22:20:22.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 -- 172.21.15.2:0/1003700704 wait complete. 2024-02-16T22:20:22.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 Processor -- start 2024-02-16T22:20:22.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 -- start start 2024-02-16T22:20:22.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 0x7fa9ac10db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:22.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac0fe0e0 0x7fa9ac10e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:22.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac10e6d0 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b3332700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac10ef10 con 0x7fa9ac0fe0e0 2024-02-16T22:20:22.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b10ce700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 0x7fa9ac10db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b10ce700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 0x7fa9ac10db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39308/0 (socket says 172.21.15.2:39308) 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.426+0000 7fa9b10ce700 1 -- 172.21.15.2:0/2169887244 learned_addr learned my addr 172.21.15.2:0/2169887244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9b08cd700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac0fe0e0 0x7fa9ac10e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9b10ce700 1 -- 172.21.15.2:0/2169887244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac0fe0e0 msgr2=0x7fa9ac10e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9b10ce700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac0fe0e0 0x7fa9ac10e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9b10ce700 1 -- 172.21.15.2:0/2169887244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a8009940 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9b10ce700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 0x7fa9ac10db00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8000f80 tx=0x7fa9a8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9a1ffb700 1 -- 172.21.15.2:0/2169887244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9a800e040 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9a1ffb700 1 -- 172.21.15.2:0/2169887244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9a80128e0 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac10f190 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9a1ffb700 1 -- 172.21.15.2:0/2169887244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9a801c940 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.427+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac10f6d0 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.428+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac04bed0 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.429+0000 7fa9a1ffb700 1 -- 172.21.15.2:0/2169887244 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa9a8012a50 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.429+0000 7fa9a1ffb700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9980692a0 0x7fa99806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:22.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.429+0000 7fa9a1ffb700 1 -- 172.21.15.2:0/2169887244 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa9a8088f50 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.429+0000 7fa9b08cd700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9980692a0 0x7fa99806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:22.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.430+0000 7fa9b08cd700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9980692a0 0x7fa99806b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009bf0 tx=0x7fa99c0091a0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:22.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.434+0000 7fa9a1ffb700 1 -- 172.21.15.2:0/2169887244 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa9a80babd0 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.720+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa9ac105ab0 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.720+0000 7fa9a1ffb700 1 -- 172.21.15.2:0/2169887244 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fa9a8058c40 con 0x7fa9ac0fcef0 2024-02-16T22:20:22.722 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-16T22:20:22.722 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-16T22:20:22.722 INFO:teuthology.orchestra.run.smithi002.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-16T22:20:22.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9980692a0 msgr2=0x7fa99806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9980692a0 0x7fa99806b750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009bf0 tx=0x7fa99c0091a0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 msgr2=0x7fa9ac10db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 0x7fa9ac10db00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8000f80 tx=0x7fa9a8004d10 comp rx=0 tx=0).stop 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 shutdown_connections 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fa9980692a0 0x7fa99806b750 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9ac0fcef0 0x7fa9ac10db00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 --2- 172.21.15.2:0/2169887244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac0fe0e0 0x7fa9ac10e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 >> 172.21.15.2:0/2169887244 conn(0x7fa9ac0f8620 msgr2=0x7fa9ac1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 shutdown_connections 2024-02-16T22:20:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:22.723+0000 7fa9b3332700 1 -- 172.21.15.2:0/2169887244 wait complete. 2024-02-16T22:20:22.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:22 smithi002 ceph-mon[29154]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:22.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:22 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2853226226' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:20:22.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:22 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2169887244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-16T22:20:23.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:22 smithi103 ceph-mon[36259]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:23.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:22 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2853226226' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:20:23.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:22 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2169887244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-16T22:20:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:23 smithi002 ceph-mon[29154]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:24.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:23 smithi002 ceph-mon[29154]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-16T22:20:24.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:20:24.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:23 smithi103 ceph-mon[36259]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:24.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:23 smithi103 ceph-mon[36259]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-16T22:20:24.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:20:26.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:25 smithi002 ceph-mon[29154]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-02-16T22:20:26.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:25 smithi103 ceph-mon[36259]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-02-16T22:20:28.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:27 smithi002 ceph-mon[29154]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-16T22:20:28.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:27 smithi103 ceph-mon[36259]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-16T22:20:30.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:29 smithi002 ceph-mon[29154]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:30.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:29 smithi103 ceph-mon[36259]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:32.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:31 smithi002 ceph-mon[29154]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:32.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:31 smithi103 ceph-mon[36259]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:33.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:32 smithi002 ceph-mon[29154]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:33.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:32 smithi103 ceph-mon[36259]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:36.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:35 smithi002 ceph-mon[29154]: pgmap v227: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:36.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:35 smithi103 ceph-mon[36259]: pgmap v227: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:38.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:37 smithi002 ceph-mon[29154]: pgmap v228: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:38.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:37 smithi103 ceph-mon[36259]: pgmap v228: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:39.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:38 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:20:39.226 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:38 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:20:40.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:39 smithi002 ceph-mon[29154]: pgmap v229: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:40.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:39 smithi103 ceph-mon[36259]: pgmap v229: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:41.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:40 smithi002 ceph-mon[29154]: pgmap v230: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:41.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:40 smithi103 ceph-mon[36259]: pgmap v230: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:43.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:43 smithi002 ceph-mon[29154]: pgmap v231: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:44.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:43 smithi103 ceph-mon[36259]: pgmap v231: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:46.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:45 smithi002 ceph-mon[29154]: pgmap v232: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:46.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:45 smithi103 ceph-mon[36259]: pgmap v232: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:48.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:47 smithi002 ceph-mon[29154]: pgmap v233: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:48.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:47 smithi103 ceph-mon[36259]: pgmap v233: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:50.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:49 smithi002 ceph-mon[29154]: pgmap v234: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:50.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:49 smithi103 ceph-mon[36259]: pgmap v234: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:52.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:51 smithi002 ceph-mon[29154]: pgmap v235: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:52.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:51 smithi103 ceph-mon[36259]: pgmap v235: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:52.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.877+0000 7f733cc4b700 1 -- 172.21.15.2:0/1261961233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 msgr2=0x7f73380fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:52.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.877+0000 7f733cc4b700 1 --2- 172.21.15.2:0/1261961233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 0x7f73380fe700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f732c004940 tx=0x7f732c009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:52.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.877+0000 7f733cc4b700 1 -- 172.21.15.2:0/1261961233 shutdown_connections 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.877+0000 7f733cc4b700 1 --2- 172.21.15.2:0/1261961233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 0x7f73380fe700 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.877+0000 7f733cc4b700 1 --2- 172.21.15.2:0/1261961233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73380fb220 0x7f73380fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.877+0000 7f733cc4b700 1 -- 172.21.15.2:0/1261961233 >> 172.21.15.2:0/1261961233 conn(0x7f73380f6ca0 msgr2=0x7f73380f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.878+0000 7f733cc4b700 1 -- 172.21.15.2:0/1261961233 shutdown_connections 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.878+0000 7f733cc4b700 1 -- 172.21.15.2:0/1261961233 wait complete. 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.878+0000 7f733cc4b700 1 Processor -- start 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.878+0000 7f733cc4b700 1 -- start start 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.878+0000 7f733cc4b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73380fb220 0x7f733810d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.878+0000 7f733cc4b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 0x7f733810dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.878+0000 7f733cc4b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733810e4c0 con 0x7f73380fe2b0 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.878+0000 7f733cc4b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733810ed60 con 0x7f73380fb220 2024-02-16T22:20:52.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f7335d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 0x7f733810dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f7335d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 0x7f733810dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42420/0 (socket says 172.21.15.2:42420) 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f7335d9b700 1 -- 172.21.15.2:0/3274857741 learned_addr learned my addr 172.21.15.2:0/3274857741 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f733659c700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73380fb220 0x7f733810d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f7335d9b700 1 -- 172.21.15.2:0/3274857741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73380fb220 msgr2=0x7f733810d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f7335d9b700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73380fb220 0x7f733810d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f7335d9b700 1 -- 172.21.15.2:0/3274857741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c005040 con 0x7f73380fe2b0 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f7335d9b700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 0x7f733810dec0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f732c009180 tx=0x7f732c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f7326ffd700 1 -- 172.21.15.2:0/3274857741 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f732c00e040 con 0x7f73380fe2b0 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733810efe0 con 0x7f73380fe2b0 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.879+0000 7f7326ffd700 1 -- 172.21.15.2:0/3274857741 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f732c009410 con 0x7f73380fe2b0 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.880+0000 7f7326ffd700 1 -- 172.21.15.2:0/3274857741 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f732c01c630 con 0x7f73380fe2b0 2024-02-16T22:20:52.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.880+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733810f520 con 0x7f73380fe2b0 2024-02-16T22:20:52.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.881+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733804bed0 con 0x7f73380fe2b0 2024-02-16T22:20:52.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.881+0000 7f7326ffd700 1 -- 172.21.15.2:0/3274857741 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f732c00bbf0 con 0x7f73380fe2b0 2024-02-16T22:20:52.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.881+0000 7f7326ffd700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f73200692a0 0x7f732006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:52.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.882+0000 7f7326ffd700 1 -- 172.21.15.2:0/3274857741 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f732c088510 con 0x7f73380fe2b0 2024-02-16T22:20:52.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.882+0000 7f733659c700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f73200692a0 0x7f732006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:52.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.882+0000 7f733659c700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f73200692a0 0x7f732006b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7328009fd0 tx=0x7f732800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:52.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:52.886+0000 7f7326ffd700 1 -- 172.21.15.2:0/3274857741 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f732c058280 con 0x7f73380fe2b0 2024-02-16T22:20:53.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.123+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f733810f950 con 0x7f73200692a0 2024-02-16T22:20:53.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.124+0000 7f7326ffd700 1 -- 172.21.15.2:0/3274857741 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f733810f950 con 0x7f73200692a0 2024-02-16T22:20:53.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.126+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f73200692a0 msgr2=0x7f732006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.126+0000 7f733cc4b700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f73200692a0 0x7f732006b750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7328009fd0 tx=0x7f732800b040 comp rx=0 tx=0).stop 2024-02-16T22:20:53.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.126+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 msgr2=0x7f733810dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.126+0000 7f733cc4b700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 0x7f733810dec0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f732c009180 tx=0x7f732c009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:53.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.127+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 shutdown_connections 2024-02-16T22:20:53.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.127+0000 7f733cc4b700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f73200692a0 0x7f732006b750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.127+0000 7f733cc4b700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f73380fb220 0x7f733810d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.127+0000 7f733cc4b700 1 --2- 172.21.15.2:0/3274857741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73380fe2b0 0x7f733810dec0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.127+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 >> 172.21.15.2:0/3274857741 conn(0x7f73380f6ca0 msgr2=0x7f73381014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:53.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.127+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 shutdown_connections 2024-02-16T22:20:53.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.127+0000 7f733cc4b700 1 -- 172.21.15.2:0/3274857741 wait complete. 2024-02-16T22:20:53.141 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-16T22:20:53.142 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:52 smithi002 ceph-mon[29154]: pgmap v236: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:53.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.287+0000 7f3d03be0700 1 -- 172.21.15.2:0/1172321577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 msgr2=0x7f3cfc0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.287+0000 7f3d03be0700 1 --2- 172.21.15.2:0/1172321577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 0x7f3cfc0fd420 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3cec004940 tx=0x7f3cec009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.287+0000 7f3d03be0700 1 -- 172.21.15.2:0/1172321577 shutdown_connections 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.287+0000 7f3d03be0700 1 --2- 172.21.15.2:0/1172321577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 0x7f3cfc0fd420 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.287+0000 7f3d03be0700 1 --2- 172.21.15.2:0/1172321577 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cfc0fbe40 0x7f3cfc0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.287+0000 7f3d03be0700 1 -- 172.21.15.2:0/1172321577 >> 172.21.15.2:0/1172321577 conn(0x7f3cfc0f74e0 msgr2=0x7f3cfc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.287+0000 7f3d03be0700 1 -- 172.21.15.2:0/1172321577 shutdown_connections 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.287+0000 7f3d03be0700 1 -- 172.21.15.2:0/1172321577 wait complete. 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d03be0700 1 Processor -- start 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d03be0700 1 -- start start 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d03be0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cfc0fbe40 0x7f3cfc0fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d03be0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 0x7f3cfc0fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d03be0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0fe980 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d03be0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0feaf0 con 0x7f3cfc0fbe40 2024-02-16T22:20:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d0117b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 0x7f3cfc0fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d0117b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 0x7f3cfc0fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42436/0 (socket says 172.21.15.2:42436) 2024-02-16T22:20:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d0117b700 1 -- 172.21.15.2:0/846245597 learned_addr learned my addr 172.21.15.2:0/846245597 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.288+0000 7f3d0197c700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cfc0fbe40 0x7f3cfc0fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.289+0000 7f3d0117b700 1 -- 172.21.15.2:0/846245597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cfc0fbe40 msgr2=0x7f3cfc0fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.289+0000 7f3d0117b700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cfc0fbe40 0x7f3cfc0fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.289+0000 7f3d0117b700 1 -- 172.21.15.2:0/846245597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec005040 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.289+0000 7f3d0117b700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 0x7f3cfc0fe380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3cec000c00 tx=0x7f3cec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.289+0000 7f3cf27fc700 1 -- 172.21.15.2:0/846245597 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cec021070 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.289+0000 7f3cf27fc700 1 -- 172.21.15.2:0/846245597 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cec01c400 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.289+0000 7f3cf27fc700 1 -- 172.21.15.2:0/846245597 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cec00e070 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.289+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc0fed70 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.289+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc0ff200 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.291+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc04bed0 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.291+0000 7f3cf27fc700 1 -- 172.21.15.2:0/846245597 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3cec01ca50 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.292+0000 7f3cf27fc700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3ce80692a0 0x7f3ce806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.292+0000 7f3cf27fc700 1 -- 172.21.15.2:0/846245597 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3cec0883e0 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.292+0000 7f3d0197c700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3ce80692a0 0x7f3ce806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:53.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.293+0000 7f3d0197c700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3ce80692a0 0x7f3ce806b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0ffc90 tx=0x7f3cf800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:53.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.297+0000 7f3cf27fc700 1 -- 172.21.15.2:0/846245597 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3cec058150 con 0x7f3cfc0fcfb0 2024-02-16T22:20:53.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:52 smithi103 ceph-mon[36259]: pgmap v236: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:53.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.513+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cfc065dd0 con 0x7f3ce80692a0 2024-02-16T22:20:53.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.515+0000 7f3cf27fc700 1 -- 172.21.15.2:0/846245597 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f3cfc065dd0 con 0x7f3ce80692a0 2024-02-16T22:20:53.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.516+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3ce80692a0 msgr2=0x7f3ce806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.516+0000 7f3d03be0700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3ce80692a0 0x7f3ce806b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0ffc90 tx=0x7f3cf800b040 comp rx=0 tx=0).stop 2024-02-16T22:20:53.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.516+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 msgr2=0x7f3cfc0fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.516+0000 7f3d03be0700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 0x7f3cfc0fe380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3cec000c00 tx=0x7f3cec009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:53.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.517+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 shutdown_connections 2024-02-16T22:20:53.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.517+0000 7f3d03be0700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3ce80692a0 0x7f3ce806b750 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.517+0000 7f3d03be0700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cfc0fbe40 0x7f3cfc0fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.517+0000 7f3d03be0700 1 --2- 172.21.15.2:0/846245597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc0fcfb0 0x7f3cfc0fe380 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.517+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 >> 172.21.15.2:0/846245597 conn(0x7f3cfc0f74e0 msgr2=0x7f3cfc0656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:53.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.517+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 shutdown_connections 2024-02-16T22:20:53.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.517+0000 7f3d03be0700 1 -- 172.21.15.2:0/846245597 wait complete. 2024-02-16T22:20:53.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.621+0000 7f3cd17bc700 1 -- 172.21.15.2:0/3475933207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fc7f0 msgr2=0x7f3ccc0fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.621+0000 7f3cd17bc700 1 --2- 172.21.15.2:0/3475933207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fc7f0 0x7f3ccc0fcc40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4004940 tx=0x7f3cb4009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:53.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.621+0000 7f3cd17bc700 1 -- 172.21.15.2:0/3475933207 shutdown_connections 2024-02-16T22:20:53.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.621+0000 7f3cd17bc700 1 --2- 172.21.15.2:0/3475933207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fc7f0 0x7f3ccc0fcc40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.621+0000 7f3cd17bc700 1 --2- 172.21.15.2:0/3475933207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ccc0fb5e0 0x7f3ccc0fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.621+0000 7f3cd17bc700 1 -- 172.21.15.2:0/3475933207 >> 172.21.15.2:0/3475933207 conn(0x7f3ccc0f6ca0 msgr2=0x7f3ccc0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:53.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.621+0000 7f3cd17bc700 1 -- 172.21.15.2:0/3475933207 shutdown_connections 2024-02-16T22:20:53.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.621+0000 7f3cd17bc700 1 -- 172.21.15.2:0/3475933207 wait complete. 2024-02-16T22:20:53.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3cd17bc700 1 Processor -- start 2024-02-16T22:20:53.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3cd17bc700 1 -- start start 2024-02-16T22:20:53.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3cd17bc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fb5e0 0x7f3ccc10d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:53.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3cd17bc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ccc0fc7f0 0x7f3ccc10ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:53.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3cd17bc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc10e4d0 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3cd17bc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc10ed70 con 0x7f3ccc0fc7f0 2024-02-16T22:20:53.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3ccaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fb5e0 0x7f3ccc10d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3ccaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fb5e0 0x7f3ccc10d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42458/0 (socket says 172.21.15.2:42458) 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3ccaffd700 1 -- 172.21.15.2:0/1056543876 learned_addr learned my addr 172.21.15.2:0/1056543876 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3cca7fc700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ccc0fc7f0 0x7f3ccc10ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3ccaffd700 1 -- 172.21.15.2:0/1056543876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ccc0fc7f0 msgr2=0x7f3ccc10ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3ccaffd700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ccc0fc7f0 0x7f3ccc10ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.622+0000 7f3ccaffd700 1 -- 172.21.15.2:0/1056543876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb4005040 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.623+0000 7f3ccaffd700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fb5e0 0x7f3ccc10d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00eb70 tx=0x7f3cbc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.623+0000 7f3cc37fe700 1 -- 172.21.15.2:0/1056543876 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cbc007cc0 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.623+0000 7f3cc37fe700 1 -- 172.21.15.2:0/1056543876 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cbc00f040 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.623+0000 7f3cc37fe700 1 -- 172.21.15.2:0/1056543876 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cbc00cb20 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.623+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ccc10eff0 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.623+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc10f530 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.624+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc04bed0 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.624+0000 7f3cc37fe700 1 -- 172.21.15.2:0/1056543876 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f3cbc00cc80 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.625+0000 7f3cc37fe700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3cb80692a0 0x7f3cb806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:53.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.625+0000 7f3cc37fe700 1 -- 172.21.15.2:0/1056543876 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3cbc086b20 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.625+0000 7f3cca7fc700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3cb80692a0 0x7f3cb806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:53.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.625+0000 7f3cca7fc700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3cb80692a0 0x7f3cb806b750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400b0d0 tx=0x7f3cb4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:53.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.629+0000 7f3cc37fe700 1 -- 172.21.15.2:0/1056543876 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3cbc04dbd0 con 0x7f3ccc0fb5e0 2024-02-16T22:20:53.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.856+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ccc10f9d0 con 0x7f3cb80692a0 2024-02-16T22:20:53.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.866+0000 7f3cc37fe700 1 -- 172.21.15.2:0/1056543876 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f3ccc10f9d0 con 0x7f3cb80692a0 2024-02-16T22:20:53.868 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-16T22:20:53.868 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.smithi002 smithi002 *:9093,9094 running (6m) 91s ago 7m 25.0M - 0.25.0 c8568f914cd2 2b7780bcc981 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi002 smithi002 running (7m) 91s ago 7m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 c1d597f23cdd 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi103 smithi103 running (6m) 96s ago 6m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 db2a59e09a51 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.smithi002 smithi002 *:3000 running (5m) 91s ago 7m 87.3M - 9.4.7 954c08fa6188 14099b699363 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi002.hhufhl smithi002 running (110s) 91s ago 109s 15.3M - 17.2.7-513-gb88fae66 870a0944ccb8 12d8ab360b57 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi002.llkqse smithi002 running (117s) 91s ago 117s 10.5M - 17.2.7-513-gb88fae66 870a0944ccb8 8e266571411c 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi103.epsgnr smithi103 running (107s) 96s ago 106s 14.5M - 17.2.7-513-gb88fae66 870a0944ccb8 d43d0b5d182c 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi103.kbizjo smithi103 running (115s) 96s ago 114s 14.6M - 17.2.7-513-gb88fae66 870a0944ccb8 86d4a2e35372 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi002.resmjv smithi002 *:9283 running (8m) 91s ago 8m 480M - 17.2.7-513-gb88fae66 870a0944ccb8 f9b132a0aa69 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi103.ydydbo smithi103 *:8443,9283 running (6m) 96s ago 6m 433M - 17.2.7-513-gb88fae66 870a0944ccb8 463897d4efc5 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi002 smithi002 running (8m) 91s ago 8m 45.2M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 cc6b0b4bb587 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi103 smithi103 running (6m) 96s ago 6m 35.6M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 3d198bee2a92 2024-02-16T22:20:53.869 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi002 smithi002 *:9100 running (7m) 91s ago 7m 15.9M - 1.5.0 0da6a335fe13 e6146a57c445 2024-02-16T22:20:53.870 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi103 smithi103 *:9100 running (6m) 96s ago 6m 17.0M - 1.5.0 0da6a335fe13 0b21b215b785 2024-02-16T22:20:53.870 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (5m) 91s ago 5m 48.7M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 a3f19d068baf 2024-02-16T22:20:53.870 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (5m) 91s ago 5m 46.1M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 58697cab97b0 2024-02-16T22:20:53.870 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (4m) 91s ago 4m 47.1M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 634798c2dbad 2024-02-16T22:20:53.870 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi103 running (4m) 96s ago 4m 43.9M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 a5332c698672 2024-02-16T22:20:53.870 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi103 running (3m) 96s ago 3m 42.5M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 fa2a12f4b710 2024-02-16T22:20:53.870 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi103 running (3m) 96s ago 3m 42.1M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 175f4eefd1f6 2024-02-16T22:20:53.870 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.smithi002 smithi002 *:9095 running (5m) 91s ago 6m 55.1M - 2.43.0 a07b618ecd1d d03f44725127 2024-02-16T22:20:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3cb80692a0 msgr2=0x7f3cb806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3cb80692a0 0x7f3cb806b750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400b0d0 tx=0x7f3cb4009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fb5e0 msgr2=0x7f3ccc10d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fb5e0 0x7f3ccc10d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00eb70 tx=0x7f3cbc00b040 comp rx=0 tx=0).stop 2024-02-16T22:20:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 shutdown_connections 2024-02-16T22:20:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f3cb80692a0 0x7f3cb806b750 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ccc0fb5e0 0x7f3ccc10d990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 --2- 172.21.15.2:0/1056543876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ccc0fc7f0 0x7f3ccc10ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 >> 172.21.15.2:0/1056543876 conn(0x7f3ccc0f6ca0 msgr2=0x7f3ccc101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:53.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 shutdown_connections 2024-02-16T22:20:53.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.869+0000 7f3cd17bc700 1 -- 172.21.15.2:0/1056543876 wait complete. 2024-02-16T22:20:53.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 -- 172.21.15.2:0/3195050626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fcfb0 msgr2=0x7f51ac0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 --2- 172.21.15.2:0/3195050626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fcfb0 0x7f51ac0fd420 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f51a0004940 tx=0x7f51a0009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:53.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 -- 172.21.15.2:0/3195050626 shutdown_connections 2024-02-16T22:20:53.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 --2- 172.21.15.2:0/3195050626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fcfb0 0x7f51ac0fd420 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 --2- 172.21.15.2:0/3195050626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51ac0fbe40 0x7f51ac0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 -- 172.21.15.2:0/3195050626 >> 172.21.15.2:0/3195050626 conn(0x7f51ac0f7520 msgr2=0x7f51ac0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:53.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 -- 172.21.15.2:0/3195050626 shutdown_connections 2024-02-16T22:20:53.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 -- 172.21.15.2:0/3195050626 wait complete. 2024-02-16T22:20:53.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 Processor -- start 2024-02-16T22:20:53.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.972+0000 7f51b1047700 1 -- start start 2024-02-16T22:20:53.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51b1047700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fbe40 0x7f51ac10d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:53.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51b1047700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51ac0fcfb0 0x7f51ac10dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:53.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51b1047700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac10e3e0 con 0x7f51ac0fbe40 2024-02-16T22:20:53.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51b1047700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac10ec80 con 0x7f51ac0fcfb0 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51aad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fbe40 0x7f51ac10d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51aad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fbe40 0x7f51ac10d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42476/0 (socket says 172.21.15.2:42476) 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51aad9d700 1 -- 172.21.15.2:0/856707145 learned_addr learned my addr 172.21.15.2:0/856707145 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51aa59c700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51ac0fcfb0 0x7f51ac10dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51aad9d700 1 -- 172.21.15.2:0/856707145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51ac0fcfb0 msgr2=0x7f51ac10dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51aad9d700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51ac0fcfb0 0x7f51ac10dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51aad9d700 1 -- 172.21.15.2:0/856707145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a0005040 con 0x7f51ac0fbe40 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51aad9d700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fbe40 0x7f51ac10d8a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f519c00ec90 tx=0x7f519c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51937fe700 1 -- 172.21.15.2:0/856707145 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519c007b30 con 0x7f51ac0fbe40 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51937fe700 1 -- 172.21.15.2:0/856707145 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f519c00f040 con 0x7f51ac0fbe40 2024-02-16T22:20:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ac10ef60 con 0x7f51ac0fbe40 2024-02-16T22:20:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51937fe700 1 -- 172.21.15.2:0/856707145 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519c00cc80 con 0x7f51ac0fbe40 2024-02-16T22:20:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.973+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac10f4a0 con 0x7f51ac0fbe40 2024-02-16T22:20:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.974+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ac04bed0 con 0x7f51ac0fbe40 2024-02-16T22:20:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.975+0000 7f51937fe700 1 -- 172.21.15.2:0/856707145 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f519c007c90 con 0x7f51ac0fbe40 2024-02-16T22:20:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.975+0000 7f51937fe700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f51940692a0 0x7f519406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.975+0000 7f51937fe700 1 -- 172.21.15.2:0/856707145 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f519c0875d0 con 0x7f51ac0fbe40 2024-02-16T22:20:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.975+0000 7f51aa59c700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f51940692a0 0x7f519406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.976+0000 7f51aa59c700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f51940692a0 0x7f519406b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f51a0009b30 tx=0x7f51a0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:53.980+0000 7f51937fe700 1 -- 172.21.15.2:0/856707145 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f519c057340 con 0x7f51ac0fbe40 2024-02-16T22:20:54.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:53 smithi002 ceph-mon[29154]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:54.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:53 smithi002 ceph-mon[29154]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:54.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:53 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:20:54.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:53 smithi002 ceph-mon[29154]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:54.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.277+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f51ac10f910 con 0x7f51ac0fbe40 2024-02-16T22:20:54.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.278+0000 7f51937fe700 1 -- 172.21.15.2:0/856707145 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f519c056ed0 con 0x7f51ac0fbe40 2024-02-16T22:20:54.280 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-16T22:20:54.280 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-16T22:20:54.280 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 2 2024-02-16T22:20:54.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:20:54.280 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-16T22:20:54.280 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 2 2024-02-16T22:20:54.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:20:54.280 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-16T22:20:54.280 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 6 2024-02-16T22:20:54.281 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:20:54.281 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": { 2024-02-16T22:20:54.281 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 4 2024-02-16T22:20:54.281 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:20:54.281 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-16T22:20:54.281 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 14 2024-02-16T22:20:54.281 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-16T22:20:54.281 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-16T22:20:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f51940692a0 msgr2=0x7f519406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f51940692a0 0x7f519406b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f51a0009b30 tx=0x7f51a0009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fbe40 msgr2=0x7f51ac10d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fbe40 0x7f51ac10d8a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f519c00ec90 tx=0x7f519c00b040 comp rx=0 tx=0).stop 2024-02-16T22:20:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 shutdown_connections 2024-02-16T22:20:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f51940692a0 0x7f519406b750 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ac0fbe40 0x7f51ac10d8a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 --2- 172.21.15.2:0/856707145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51ac0fcfb0 0x7f51ac10dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 >> 172.21.15.2:0/856707145 conn(0x7f51ac0f7520 msgr2=0x7f51ac065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:54.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 shutdown_connections 2024-02-16T22:20:54.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.280+0000 7f51b1047700 1 -- 172.21.15.2:0/856707145 wait complete. 2024-02-16T22:20:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:53 smithi103 ceph-mon[36259]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:53 smithi103 ceph-mon[36259]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:53 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:20:54.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:53 smithi103 ceph-mon[36259]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:54.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.388+0000 7f4dc37b2700 1 -- 172.21.15.2:0/2820432668 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fcfb0 msgr2=0x7f4dbc0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.388+0000 7f4dc37b2700 1 --2- 172.21.15.2:0/2820432668 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fcfb0 0x7f4dbc0fd420 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4db8004940 tx=0x7f4db8009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:54.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.388+0000 7f4dc37b2700 1 -- 172.21.15.2:0/2820432668 shutdown_connections 2024-02-16T22:20:54.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.388+0000 7f4dc37b2700 1 --2- 172.21.15.2:0/2820432668 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fcfb0 0x7f4dbc0fd420 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.388+0000 7f4dc37b2700 1 --2- 172.21.15.2:0/2820432668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dbc0fbe40 0x7f4dbc0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.388+0000 7f4dc37b2700 1 -- 172.21.15.2:0/2820432668 >> 172.21.15.2:0/2820432668 conn(0x7f4dbc0f74e0 msgr2=0x7f4dbc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.388+0000 7f4dc37b2700 1 -- 172.21.15.2:0/2820432668 shutdown_connections 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.388+0000 7f4dc37b2700 1 -- 172.21.15.2:0/2820432668 wait complete. 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc37b2700 1 Processor -- start 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc37b2700 1 -- start start 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc37b2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fbe40 0x7f4dbc0fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc37b2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dbc0fcfb0 0x7f4dbc0fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc37b2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0fe980 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc37b2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0feaf0 con 0x7f4dbc0fcfb0 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc154e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fbe40 0x7f4dbc0fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc154e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fbe40 0x7f4dbc0fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42498/0 (socket says 172.21.15.2:42498) 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc154e700 1 -- 172.21.15.2:0/1948558777 learned_addr learned my addr 172.21.15.2:0/1948558777 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc0d4d700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dbc0fcfb0 0x7f4dbc0fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:54.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc154e700 1 -- 172.21.15.2:0/1948558777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dbc0fcfb0 msgr2=0x7f4dbc0fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc154e700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dbc0fcfb0 0x7f4dbc0fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.389+0000 7f4dc154e700 1 -- 172.21.15.2:0/1948558777 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db8005040 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.390+0000 7f4dc154e700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fbe40 0x7f4dbc0fff00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4dac002a50 tx=0x7f4dac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:54.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.390+0000 7f4db1ffb700 1 -- 172.21.15.2:0/1948558777 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dac007cc0 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.390+0000 7f4db1ffb700 1 -- 172.21.15.2:0/1948558777 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4dac00f040 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.390+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dbc0fedd0 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.390+0000 7f4db1ffb700 1 -- 172.21.15.2:0/1948558777 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dac00da30 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.390+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dbc115990 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.391+0000 7f4db1ffb700 1 -- 172.21.15.2:0/1948558777 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f4dac00db90 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.391+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dbc04bed0 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.392+0000 7f4db1ffb700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4da80692a0 0x7f4da806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.392+0000 7f4db1ffb700 1 -- 172.21.15.2:0/1948558777 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4dac086f70 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.392+0000 7f4dc0d4d700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4da80692a0 0x7f4da806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:54.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.392+0000 7f4dc0d4d700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4da80692a0 0x7f4da806b750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4db800b3b0 tx=0x7f4db8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:54.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.396+0000 7f4db1ffb700 1 -- 172.21.15.2:0/1948558777 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4dac056c60 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.655+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4dbc115de0 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.656+0000 7f4db1ffb700 1 -- 172.21.15.2:0/1948558777 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1830 (secure 0 0 0) 0x7f4dac0567f0 con 0x7f4dbc0fbe40 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:e16 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 16 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-02-16T22:18:52.996722+0000 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2024-02-16T22:19:10.960547+0000 2024-02-16T22:20:54.658 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 60 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 46 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 1 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:in 0 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24263} 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3] 2024-02-16T22:20:54.659 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data enabled 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 1 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.kbizjo{0:24263} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.hhufhl{0:14486} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.2:6828/971745533,v1:172.21.15.2:6829/971745533] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.llkqse{-1:14496} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.2:6826/3769723051,v1:172.21.15.2:6827/3769723051] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:20:54.660 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.epsgnr{-1:24271} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.103:6826/1310923282,v1:172.21.15.103:6827/1310923282] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:20:54.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4da80692a0 msgr2=0x7f4da806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4da80692a0 0x7f4da806b750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4db800b3b0 tx=0x7f4db8009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:54.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fbe40 msgr2=0x7f4dbc0fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fbe40 0x7f4dbc0fff00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4dac002a50 tx=0x7f4dac0046b0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 shutdown_connections 2024-02-16T22:20:54.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f4da80692a0 0x7f4da806b750 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dbc0fbe40 0x7f4dbc0fff00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 --2- 172.21.15.2:0/1948558777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4dbc0fcfb0 0x7f4dbc0fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 >> 172.21.15.2:0/1948558777 conn(0x7f4dbc0f74e0 msgr2=0x7f4dbc0656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:54.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 shutdown_connections 2024-02-16T22:20:54.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.658+0000 7f4dc37b2700 1 -- 172.21.15.2:0/1948558777 wait complete. 2024-02-16T22:20:54.662 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 16 2024-02-16T22:20:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.765+0000 7f31b5a01700 1 -- 172.21.15.2:0/4178087040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 msgr2=0x7f31b00fe400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.765+0000 7f31b5a01700 1 --2- 172.21.15.2:0/4178087040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 0x7f31b00fe400 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f31a4004940 tx=0x7f31a4009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.765+0000 7f31b5a01700 1 -- 172.21.15.2:0/4178087040 shutdown_connections 2024-02-16T22:20:54.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.765+0000 7f31b5a01700 1 --2- 172.21.15.2:0/4178087040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 0x7f31b00fe400 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.765+0000 7f31b5a01700 1 --2- 172.21.15.2:0/4178087040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b00fcda0 0x7f31b00fd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.765+0000 7f31b5a01700 1 -- 172.21.15.2:0/4178087040 >> 172.21.15.2:0/4178087040 conn(0x7f31b00f8480 msgr2=0x7f31b00fa890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:54.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.765+0000 7f31b5a01700 1 -- 172.21.15.2:0/4178087040 shutdown_connections 2024-02-16T22:20:54.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.765+0000 7f31b5a01700 1 -- 172.21.15.2:0/4178087040 wait complete. 2024-02-16T22:20:54.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.766+0000 7f31b5a01700 1 Processor -- start 2024-02-16T22:20:54.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.766+0000 7f31b5a01700 1 -- start start 2024-02-16T22:20:54.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31b5a01700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b00fcda0 0x7f31b010d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31b5a01700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 0x7f31b010dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31b5a01700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b010e1d0 con 0x7f31b00fdfb0 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31b5a01700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b010ea70 con 0x7f31b00fcda0 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31ae7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 0x7f31b010dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31aeffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b00fcda0 0x7f31b010d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31ae7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 0x7f31b010dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42508/0 (socket says 172.21.15.2:42508) 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31ae7fc700 1 -- 172.21.15.2:0/3712688765 learned_addr learned my addr 172.21.15.2:0/3712688765 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31ae7fc700 1 -- 172.21.15.2:0/3712688765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b00fcda0 msgr2=0x7f31b010d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31ae7fc700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b00fcda0 0x7f31b010d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31ae7fc700 1 -- 172.21.15.2:0/3712688765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a4005040 con 0x7f31b00fdfb0 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31ae7fc700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 0x7f31b010dbd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009b30 tx=0x7f31a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31977fe700 1 -- 172.21.15.2:0/3712688765 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31a4021070 con 0x7f31b00fdfb0 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b010ecf0 con 0x7f31b00fdfb0 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31977fe700 1 -- 172.21.15.2:0/3712688765 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31a401c400 con 0x7f31b00fdfb0 2024-02-16T22:20:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31977fe700 1 -- 172.21.15.2:0/3712688765 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31a400e070 con 0x7f31b00fdfb0 2024-02-16T22:20:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.767+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b010f1d0 con 0x7f31b00fdfb0 2024-02-16T22:20:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.768+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b004bed0 con 0x7f31b00fdfb0 2024-02-16T22:20:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.769+0000 7f31977fe700 1 -- 172.21.15.2:0/3712688765 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f31a401c970 con 0x7f31b00fdfb0 2024-02-16T22:20:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.769+0000 7f31977fe700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f31980692a0 0x7f319806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.769+0000 7f31977fe700 1 -- 172.21.15.2:0/3712688765 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f31a4088490 con 0x7f31b00fdfb0 2024-02-16T22:20:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.769+0000 7f31aeffd700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f31980692a0 0x7f319806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.770+0000 7f31aeffd700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f31980692a0 0x7f319806b750 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009b80 tx=0x7f31a000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:54.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.774+0000 7f31977fe700 1 -- 172.21.15.2:0/3712688765 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f31a4058200 con 0x7f31b00fdfb0 2024-02-16T22:20:54.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.989+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f31b010f5e0 con 0x7f31980692a0 2024-02-16T22:20:54.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.990+0000 7f31977fe700 1 -- 172.21.15.2:0/3712688765 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f31b010f5e0 con 0x7f31980692a0 2024-02-16T22:20:54.991 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-16T22:20:54.992 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", 2024-02-16T22:20:54.992 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-16T22:20:54.992 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-16T22:20:54.992 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-02-16T22:20:54.992 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "", 2024-02-16T22:20:54.992 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 image", 2024-02-16T22:20:54.992 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-16T22:20:54.992 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-16T22:20:54.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f31980692a0 msgr2=0x7f319806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f31980692a0 0x7f319806b750 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009b80 tx=0x7f31a000b040 comp rx=0 tx=0).stop 2024-02-16T22:20:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 msgr2=0x7f31b010dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 0x7f31b010dbd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009b30 tx=0x7f31a4009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 shutdown_connections 2024-02-16T22:20:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f31980692a0 0x7f319806b750 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b00fcda0 0x7f31b010d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 --2- 172.21.15.2:0/3712688765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31b00fdfb0 0x7f31b010dbd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 >> 172.21.15.2:0/3712688765 conn(0x7f31b00f8480 msgr2=0x7f31b01011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 shutdown_connections 2024-02-16T22:20:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:54.992+0000 7f31b5a01700 1 -- 172.21.15.2:0/3712688765 wait complete. 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.095+0000 7f6465c32700 1 -- 172.21.15.2:0/3760282756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 msgr2=0x7f64600fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.095+0000 7f6465c32700 1 --2- 172.21.15.2:0/3760282756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 0x7f64600fd420 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6454004940 tx=0x7f6454009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.095+0000 7f6465c32700 1 -- 172.21.15.2:0/3760282756 shutdown_connections 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.095+0000 7f6465c32700 1 --2- 172.21.15.2:0/3760282756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 0x7f64600fd420 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.095+0000 7f6465c32700 1 --2- 172.21.15.2:0/3760282756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64600fbe40 0x7f64600fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.095+0000 7f6465c32700 1 -- 172.21.15.2:0/3760282756 >> 172.21.15.2:0/3760282756 conn(0x7f64600f74e0 msgr2=0x7f64600f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.096+0000 7f6465c32700 1 -- 172.21.15.2:0/3760282756 shutdown_connections 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.096+0000 7f6465c32700 1 -- 172.21.15.2:0/3760282756 wait complete. 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.096+0000 7f6465c32700 1 Processor -- start 2024-02-16T22:20:55.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.096+0000 7f6465c32700 1 -- start start 2024-02-16T22:20:55.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f6465c32700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64600fbe40 0x7f646010d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:55.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f6465c32700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 0x7f646010ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f6465c32700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646010e3f0 con 0x7f64600fcfb0 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f6465c32700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646010ec90 con 0x7f64600fbe40 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 0x7f646010ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64600fbe40 0x7f646010d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 0x7f646010ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42530/0 (socket says 172.21.15.2:42530) 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645effd700 1 -- 172.21.15.2:0/696952188 learned_addr learned my addr 172.21.15.2:0/696952188 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645f7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64600fbe40 0x7f646010d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.2:44498/0 (socket says 172.21.15.2:44498) 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645f7fe700 1 -- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 msgr2=0x7f646010ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645f7fe700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 0x7f646010ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645f7fe700 1 -- 172.21.15.2:0/696952188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454005040 con 0x7f64600fbe40 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645effd700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 0x7f646010ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f645f7fe700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64600fbe40 0x7f646010d8b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6450009fd0 tx=0x7f645000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.097+0000 7f6447fff700 1 -- 172.21.15.2:0/696952188 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6450004e90 con 0x7f64600fbe40 2024-02-16T22:20:55.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.098+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646010ef10 con 0x7f64600fbe40 2024-02-16T22:20:55.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.098+0000 7f6447fff700 1 -- 172.21.15.2:0/696952188 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f645000f040 con 0x7f64600fbe40 2024-02-16T22:20:55.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.098+0000 7f6447fff700 1 -- 172.21.15.2:0/696952188 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f645000ca50 con 0x7f64600fbe40 2024-02-16T22:20:55.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.098+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646010f450 con 0x7f64600fbe40 2024-02-16T22:20:55.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.098+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646004bed0 con 0x7f64600fbe40 2024-02-16T22:20:55.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.099+0000 7f6447fff700 1 -- 172.21.15.2:0/696952188 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f6450007cc0 con 0x7f64600fbe40 2024-02-16T22:20:55.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.100+0000 7f6447fff700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64480692a0 0x7f644806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:20:55.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.100+0000 7f6447fff700 1 -- 172.21.15.2:0/696952188 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6450086c80 con 0x7f64600fbe40 2024-02-16T22:20:55.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.100+0000 7f645effd700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64480692a0 0x7f644806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:20:55.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.100+0000 7f645effd700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64480692a0 0x7f644806b750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f645400b0d0 tx=0x7f6454009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:20:55.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.107+0000 7f6447fff700 1 -- 172.21.15.2:0/696952188 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6450056a00 con 0x7f64600fbe40 2024-02-16T22:20:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:54 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/856707145' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-16T22:20:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:54 smithi002 ceph-mon[29154]: pgmap v237: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:54 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1948558777' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:20:55.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:54 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/856707145' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-16T22:20:55.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:54 smithi103 ceph-mon[36259]: pgmap v237: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:20:55.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:54 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1948558777' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:20:55.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.383+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f646010f670 con 0x7f64600fbe40 2024-02-16T22:20:55.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.384+0000 7f6447fff700 1 -- 172.21.15.2:0/696952188 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f6450056590 con 0x7f64600fbe40 2024-02-16T22:20:55.386 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-16T22:20:55.386 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-16T22:20:55.386 INFO:teuthology.orchestra.run.smithi002.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64480692a0 msgr2=0x7f644806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64480692a0 0x7f644806b750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f645400b0d0 tx=0x7f6454009e30 comp rx=0 tx=0).stop 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64600fbe40 msgr2=0x7f646010d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64600fbe40 0x7f646010d8b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6450009fd0 tx=0x7f645000b040 comp rx=0 tx=0).stop 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 shutdown_connections 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64480692a0 0x7f644806b750 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64600fbe40 0x7f646010d8b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 --2- 172.21.15.2:0/696952188 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64600fcfb0 0x7f646010ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 >> 172.21.15.2:0/696952188 conn(0x7f64600f74e0 msgr2=0x7f64600656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 shutdown_connections 2024-02-16T22:20:55.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:20:55.386+0000 7f6465c32700 1 -- 172.21.15.2:0/696952188 wait complete. 2024-02-16T22:20:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:55 smithi002 ceph-mon[29154]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:56.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:55 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/696952188' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-16T22:20:56.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:55 smithi103 ceph-mon[36259]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:20:56.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:55 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/696952188' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-16T22:20:57.186 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:56 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:20:57.186 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:56 smithi002 ceph-mon[29154]: Upgrade: Target is version 19.0.0-1450-gf78a58c0 (unknown) 2024-02-16T22:20:57.187 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:56 smithi002 ceph-mon[29154]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, digests ['quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526'] 2024-02-16T22:20:57.187 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:56 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:20:57.187 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:56 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:20:57.187 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:56 smithi002 ceph-mon[29154]: Upgrade: Need to upgrade myself (mgr.smithi002.resmjv) 2024-02-16T22:20:57.187 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:56 smithi002 ceph-mon[29154]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 on smithi103 2024-02-16T22:20:57.187 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:56 smithi002 ceph-mon[29154]: pgmap v238: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:57.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:56 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:20:57.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:56 smithi103 ceph-mon[36259]: Upgrade: Target is version 19.0.0-1450-gf78a58c0 (unknown) 2024-02-16T22:20:57.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:56 smithi103 ceph-mon[36259]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, digests ['quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526'] 2024-02-16T22:20:57.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:56 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-16T22:20:57.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:56 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' 2024-02-16T22:20:57.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:56 smithi103 ceph-mon[36259]: Upgrade: Need to upgrade myself (mgr.smithi002.resmjv) 2024-02-16T22:20:57.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:56 smithi103 ceph-mon[36259]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 on smithi103 2024-02-16T22:20:57.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:56 smithi103 ceph-mon[36259]: pgmap v238: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:20:59.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:20:59 smithi002 ceph-mon[29154]: pgmap v239: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:00.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:20:59 smithi103 ceph-mon[36259]: pgmap v239: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:02.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:01 smithi002 ceph-mon[29154]: pgmap v240: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:21:02.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:01 smithi103 ceph-mon[36259]: pgmap v240: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:21:03.915 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:03 smithi103 ceph-mon[36259]: pgmap v241: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:04.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:03 smithi002 ceph-mon[29154]: pgmap v241: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:06.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:05 smithi002 ceph-mon[29154]: pgmap v242: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:21:06.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:05 smithi103 ceph-mon[36259]: pgmap v242: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:21:07.914 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:07 smithi002 ceph-mon[29154]: pgmap v243: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:08.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:07 smithi103 ceph-mon[36259]: pgmap v243: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:09.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:08 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:21:09.225 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:08 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:21:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:09 smithi002 ceph-mon[29154]: pgmap v244: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:10.327 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:09 smithi103 ceph-mon[36259]: pgmap v244: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:11.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:10 smithi002 ceph-mon[29154]: pgmap v245: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:21:11.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:10 smithi103 ceph-mon[36259]: pgmap v245: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-16T22:21:11.576 INFO:tasks.workunit.client.1.smithi103.stderr:Note: switching to 'ed7ab725e1a43615009dfb74a42bf33d80081bc8'. 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr: 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr:state without impacting any branches by switching back to a branch. 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr: 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr: 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr: git switch -c 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr: 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr:Or undo this operation with: 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr: 2024-02-16T22:21:11.577 INFO:tasks.workunit.client.1.smithi103.stderr: git switch - 2024-02-16T22:21:11.578 INFO:tasks.workunit.client.1.smithi103.stderr: 2024-02-16T22:21:11.578 INFO:tasks.workunit.client.1.smithi103.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-16T22:21:11.578 INFO:tasks.workunit.client.1.smithi103.stderr: 2024-02-16T22:21:11.578 INFO:tasks.workunit.client.1.smithi103.stderr:HEAD is now at ed7ab725e1a qa: test fuse/kclient for mds upgrade seq 2024-02-16T22:21:11.581 DEBUG:teuthology.orchestra.run.smithi103:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-16T22:21:11.647 INFO:tasks.workunit.client.1.smithi103.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-16T22:21:11.650 INFO:tasks.workunit.client.1.smithi103.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-16T22:21:11.650 INFO:tasks.workunit.client.1.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-16T22:21:12.550 INFO:tasks.workunit.client.1.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-16T22:21:12.624 INFO:tasks.workunit.client.1.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-16T22:21:12.667 INFO:tasks.workunit.client.1.smithi103.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-16T22:21:12.669 INFO:tasks.workunit.client.1.smithi103.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-16T22:21:12.669 INFO:tasks.workunit.client.1.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-16T22:21:12.712 INFO:tasks.workunit.client.1.smithi103.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-16T22:21:12.715 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:21:12.715 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-16T22:21:12.771 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-16T22:21:12.773 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-16T22:21:12.773 DEBUG:teuthology.orchestra.run.smithi103:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed7ab725e1a43615009dfb74a42bf33d80081bc8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-16T22:21:12.832 INFO:tasks.workunit.client.1.smithi103.stderr:+ mkdir -p fsstress 2024-02-16T22:21:12.833 INFO:tasks.workunit.client.1.smithi103.stderr:+ pushd fsstress 2024-02-16T22:21:12.833 INFO:tasks.workunit.client.1.smithi103.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-16T22:21:12.834 INFO:tasks.workunit.client.1.smithi103.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-16T22:21:13.348 INFO:tasks.workunit.client.1.smithi103.stderr:+ tar xzf ltp-full.tgz 2024-02-16T22:21:13.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:13 smithi002 ceph-mon[29154]: pgmap v246: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:14.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:13 smithi103 ceph-mon[36259]: pgmap v246: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-16T22:21:16.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:15 smithi002 ceph-mon[29154]: pgmap v247: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 111 KiB/s wr, 2 op/s 2024-02-16T22:21:16.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:15 smithi103 ceph-mon[36259]: pgmap v247: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 111 KiB/s wr, 2 op/s 2024-02-16T22:21:18.226 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:17 smithi002 ceph-mon[29154]: pgmap v248: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 115 KiB/s wr, 2 op/s 2024-02-16T22:21:18.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:17 smithi103 ceph-mon[36259]: pgmap v248: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 115 KiB/s wr, 2 op/s 2024-02-16T22:21:20.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:19 smithi002 ceph-mon[29154]: pgmap v249: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 115 KiB/s wr, 2 op/s 2024-02-16T22:21:20.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:19 smithi103 ceph-mon[36259]: pgmap v249: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 115 KiB/s wr, 2 op/s 2024-02-16T22:21:21.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:20 smithi002 ceph-mon[29154]: pgmap v250: 65 pgs: 65 active+clean; 41 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 795 KiB/s rd, 3.6 MiB/s wr, 324 op/s 2024-02-16T22:21:21.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:20 smithi103 ceph-mon[36259]: pgmap v250: 65 pgs: 65 active+clean; 41 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 795 KiB/s rd, 3.6 MiB/s wr, 324 op/s 2024-02-16T22:21:23.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:23 smithi002 ceph-mon[29154]: pgmap v251: 65 pgs: 65 active+clean; 44 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 795 KiB/s rd, 3.8 MiB/s wr, 368 op/s 2024-02-16T22:21:23.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:23 smithi002 ceph-mon[29154]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:21:24.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:23 smithi103 ceph-mon[36259]: pgmap v251: 65 pgs: 65 active+clean; 44 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 795 KiB/s rd, 3.8 MiB/s wr, 368 op/s 2024-02-16T22:21:24.075 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:23 smithi103 ceph-mon[36259]: from='mgr.14212 172.21.15.2:0/1322288100' entity='mgr.smithi002.resmjv' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-16T22:21:25.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.515+0000 7f64b1bca700 1 -- 172.21.15.2:0/838262411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac1000c0 msgr2=0x7f64ac102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:25.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.515+0000 7f64b1bca700 1 --2- 172.21.15.2:0/838262411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac1000c0 0x7f64ac102520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f64a0004940 tx=0x7f64a0009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.515+0000 7f64b1bca700 1 -- 172.21.15.2:0/838262411 shutdown_connections 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.515+0000 7f64b1bca700 1 --2- 172.21.15.2:0/838262411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac1000c0 0x7f64ac102520 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.515+0000 7f64b1bca700 1 --2- 172.21.15.2:0/838262411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac066c70 0x7f64ac0ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.515+0000 7f64b1bca700 1 -- 172.21.15.2:0/838262411 >> 172.21.15.2:0/838262411 conn(0x7f64ac0f7500 msgr2=0x7f64ac0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.515+0000 7f64b1bca700 1 -- 172.21.15.2:0/838262411 shutdown_connections 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.515+0000 7f64b1bca700 1 -- 172.21.15.2:0/838262411 wait complete. 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.516+0000 7f64b1bca700 1 Processor -- start 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.516+0000 7f64b1bca700 1 -- start start 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.516+0000 7f64b1bca700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac066c70 0x7f64ac10d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.516+0000 7f64b1bca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac1000c0 0x7f64ac10de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.516+0000 7f64b1bca700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac10e410 con 0x7f64ac066c70 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.516+0000 7f64b1bca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac10ecb0 con 0x7f64ac1000c0 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.516+0000 7f64ab7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac066c70 0x7f64ac10d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:25.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.516+0000 7f64ab7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac066c70 0x7f64ac10d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39398/0 (socket says 172.21.15.2:39398) 2024-02-16T22:21:25.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.516+0000 7f64ab7fe700 1 -- 172.21.15.2:0/1795257292 learned_addr learned my addr 172.21.15.2:0/1795257292 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:21:25.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.517+0000 7f64aaffd700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac1000c0 0x7f64ac10de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:25.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.517+0000 7f64ab7fe700 1 -- 172.21.15.2:0/1795257292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac1000c0 msgr2=0x7f64ac10de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:25.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.517+0000 7f64ab7fe700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac1000c0 0x7f64ac10de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:25.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.517+0000 7f64ab7fe700 1 -- 172.21.15.2:0/1795257292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a0005040 con 0x7f64ac066c70 2024-02-16T22:21:25.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.517+0000 7f64ab7fe700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac066c70 0x7f64ac10d8d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f649c00eb40 tx=0x7f649c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:25.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.517+0000 7f6493fff700 1 -- 172.21.15.2:0/1795257292 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f649c007cc0 con 0x7f64ac066c70 2024-02-16T22:21:25.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.517+0000 7f6493fff700 1 -- 172.21.15.2:0/1795257292 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f649c00f040 con 0x7f64ac066c70 2024-02-16T22:21:25.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.517+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac10ef30 con 0x7f64ac066c70 2024-02-16T22:21:25.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.518+0000 7f6493fff700 1 -- 172.21.15.2:0/1795257292 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f649c00ca10 con 0x7f64ac066c70 2024-02-16T22:21:25.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.518+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac10f470 con 0x7f64ac066c70 2024-02-16T22:21:25.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.519+0000 7f6493fff700 1 -- 172.21.15.2:0/1795257292 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f649c00cc80 con 0x7f64ac066c70 2024-02-16T22:21:25.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.519+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac04bed0 con 0x7f64ac066c70 2024-02-16T22:21:25.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.519+0000 7f6493fff700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64940692a0 0x7f649406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:25.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.519+0000 7f6493fff700 1 -- 172.21.15.2:0/1795257292 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f649c086dd0 con 0x7f64ac066c70 2024-02-16T22:21:25.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.520+0000 7f64aaffd700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64940692a0 0x7f649406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:25.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.520+0000 7f64aaffd700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64940692a0 0x7f649406b750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f64a0000c00 tx=0x7f64a0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:25.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.527+0000 7f6493fff700 1 -- 172.21.15.2:0/1795257292 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f649c04dbd0 con 0x7f64ac066c70 2024-02-16T22:21:25.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.786+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f64ac10f8a0 con 0x7f64940692a0 2024-02-16T22:21:25.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.787+0000 7f6493fff700 1 -- 172.21.15.2:0/1795257292 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f64ac10f8a0 con 0x7f64940692a0 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64940692a0 msgr2=0x7f649406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64940692a0 0x7f649406b750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f64a0000c00 tx=0x7f64a0009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac066c70 msgr2=0x7f64ac10d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac066c70 0x7f64ac10d8d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f649c00eb40 tx=0x7f649c00b040 comp rx=0 tx=0).stop 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 shutdown_connections 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f64940692a0 0x7f649406b750 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64ac066c70 0x7f64ac10d8d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 --2- 172.21.15.2:0/1795257292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64ac1000c0 0x7f64ac10de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 >> 172.21.15.2:0/1795257292 conn(0x7f64ac0f7500 msgr2=0x7f64ac0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 shutdown_connections 2024-02-16T22:21:25.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.790+0000 7f64b1bca700 1 -- 172.21.15.2:0/1795257292 wait complete. 2024-02-16T22:21:25.806 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-02-16T22:21:25.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.900+0000 7f36c147a700 1 -- 172.21.15.2:0/60739805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36bc1003a0 msgr2=0x7f36bc102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:25.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.900+0000 7f36c147a700 1 --2- 172.21.15.2:0/60739805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36bc1003a0 0x7f36bc102780 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f36b0004940 tx=0x7f36b0009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:25.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.900+0000 7f36c147a700 1 -- 172.21.15.2:0/60739805 shutdown_connections 2024-02-16T22:21:25.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.900+0000 7f36c147a700 1 --2- 172.21.15.2:0/60739805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36bc1003a0 0x7f36bc102780 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:25.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.900+0000 7f36c147a700 1 --2- 172.21.15.2:0/60739805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36bc0fda80 0x7f36bc0ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:25.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.900+0000 7f36c147a700 1 -- 172.21.15.2:0/60739805 >> 172.21.15.2:0/60739805 conn(0x7f36bc0f7500 msgr2=0x7f36bc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:25.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.900+0000 7f36c147a700 1 -- 172.21.15.2:0/60739805 shutdown_connections 2024-02-16T22:21:25.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.900+0000 7f36c147a700 1 -- 172.21.15.2:0/60739805 wait complete. 2024-02-16T22:21:25.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36c147a700 1 Processor -- start 2024-02-16T22:21:25.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36c147a700 1 -- start start 2024-02-16T22:21:25.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36c147a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36bc0fda80 0x7f36bc10d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:25.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36c147a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36bc1003a0 0x7f36bc10de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:25.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36c147a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc10e400 con 0x7f36bc0fda80 2024-02-16T22:21:25.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36c147a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc10eca0 con 0x7f36bc1003a0 2024-02-16T22:21:25.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36ba7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36bc1003a0 0x7f36bc10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:25.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36ba7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36bc1003a0 0x7f36bc10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.2:34512/0 (socket says 172.21.15.2:34512) 2024-02-16T22:21:25.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36ba7fc700 1 -- 172.21.15.2:0/959855782 learned_addr learned my addr 172.21.15.2:0/959855782 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:21:25.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.901+0000 7f36baffd700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36bc0fda80 0x7f36bc10d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:25.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36ba7fc700 1 -- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36bc0fda80 msgr2=0x7f36bc10d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:25.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36ba7fc700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36bc0fda80 0x7f36bc10d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:25.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36ba7fc700 1 -- 172.21.15.2:0/959855782 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0005040 con 0x7f36bc1003a0 2024-02-16T22:21:25.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36baffd700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36bc0fda80 0x7f36bc10d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-16T22:21:25.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36ba7fc700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36bc1003a0 0x7f36bc10de00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f36b0000c00 tx=0x7f36b0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:25.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36a37fe700 1 -- 172.21.15.2:0/959855782 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b0021070 con 0x7f36bc1003a0 2024-02-16T22:21:25.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36bc10ef20 con 0x7f36bc1003a0 2024-02-16T22:21:25.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36a37fe700 1 -- 172.21.15.2:0/959855782 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36b001c400 con 0x7f36bc1003a0 2024-02-16T22:21:25.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36a37fe700 1 -- 172.21.15.2:0/959855782 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b000e070 con 0x7f36bc1003a0 2024-02-16T22:21:25.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.902+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc10f340 con 0x7f36bc1003a0 2024-02-16T22:21:25.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.904+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36bc107a50 con 0x7f36bc1003a0 2024-02-16T22:21:25.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.904+0000 7f36a37fe700 1 -- 172.21.15.2:0/959855782 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f36b0013af0 con 0x7f36bc1003a0 2024-02-16T22:21:25.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.905+0000 7f36a37fe700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f36a40692a0 0x7f36a406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:25.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.905+0000 7f36a37fe700 1 -- 172.21.15.2:0/959855782 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f36b0088440 con 0x7f36bc1003a0 2024-02-16T22:21:25.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.905+0000 7f36baffd700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f36a40692a0 0x7f36a406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:25.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.906+0000 7f36baffd700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f36a40692a0 0x7f36a406b750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f36ac004600 tx=0x7f36ac00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:25.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:25.913+0000 7f36a37fe700 1 -- 172.21.15.2:0/959855782 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f36b0058190 con 0x7f36bc1003a0 2024-02-16T22:21:26.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.128+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36bc02b340 con 0x7f36a40692a0 2024-02-16T22:21:26.130 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:25 smithi002 ceph-mon[29154]: pgmap v252: 65 pgs: 65 active+clean; 61 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 455 op/s 2024-02-16T22:21:26.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.130+0000 7f36a37fe700 1 -- 172.21.15.2:0/959855782 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f36bc02b340 con 0x7f36a40692a0 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f36a40692a0 msgr2=0x7f36a406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f36a40692a0 0x7f36a406b750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f36ac004600 tx=0x7f36ac00b040 comp rx=0 tx=0).stop 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36bc1003a0 msgr2=0x7f36bc10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36bc1003a0 0x7f36bc10de00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f36b0000c00 tx=0x7f36b0009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 shutdown_connections 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f36a40692a0 0x7f36a406b750 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36bc0fda80 0x7f36bc10d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 --2- 172.21.15.2:0/959855782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36bc1003a0 0x7f36bc10de00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 >> 172.21.15.2:0/959855782 conn(0x7f36bc0f7500 msgr2=0x7f36bc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:26.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 shutdown_connections 2024-02-16T22:21:26.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.132+0000 7f36c147a700 1 -- 172.21.15.2:0/959855782 wait complete. 2024-02-16T22:21:26.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.242+0000 7fe23f310700 1 -- 172.21.15.2:0/1053454152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2380fcfb0 msgr2=0x7fe2380fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:26.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.242+0000 7fe23f310700 1 --2- 172.21.15.2:0/1053454152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2380fcfb0 0x7fe2380fd420 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe228004940 tx=0x7fe228009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:26.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.242+0000 7fe23f310700 1 -- 172.21.15.2:0/1053454152 shutdown_connections 2024-02-16T22:21:26.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.242+0000 7fe23f310700 1 --2- 172.21.15.2:0/1053454152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2380fcfb0 0x7fe2380fd420 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.242+0000 7fe23f310700 1 --2- 172.21.15.2:0/1053454152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2380fbe40 0x7fe2380fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.242+0000 7fe23f310700 1 -- 172.21.15.2:0/1053454152 >> 172.21.15.2:0/1053454152 conn(0x7fe2380f7500 msgr2=0x7fe2380f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.242+0000 7fe23f310700 1 -- 172.21.15.2:0/1053454152 shutdown_connections 2024-02-16T22:21:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.242+0000 7fe23f310700 1 -- 172.21.15.2:0/1053454152 wait complete. 2024-02-16T22:21:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23f310700 1 Processor -- start 2024-02-16T22:21:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23f310700 1 -- start start 2024-02-16T22:21:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23f310700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2380fbe40 0x7fe238112960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23f310700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2380fcfb0 0x7fe238114eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23f310700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe238066c40 con 0x7fe2380fbe40 2024-02-16T22:21:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23f310700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe238066d80 con 0x7fe2380fcfb0 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23c8ab700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2380fcfb0 0x7fe238114eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23d0ac700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2380fbe40 0x7fe238112960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23c8ab700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2380fcfb0 0x7fe238114eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.2:34532/0 (socket says 172.21.15.2:34532) 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.243+0000 7fe23c8ab700 1 -- 172.21.15.2:0/2542728411 learned_addr learned my addr 172.21.15.2:0/2542728411 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.244+0000 7fe23d0ac700 1 -- 172.21.15.2:0/2542728411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2380fcfb0 msgr2=0x7fe238114eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.244+0000 7fe23d0ac700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2380fcfb0 0x7fe238114eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.244+0000 7fe23d0ac700 1 -- 172.21.15.2:0/2542728411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe228005040 con 0x7fe2380fbe40 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.244+0000 7fe23c8ab700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2380fcfb0 0x7fe238114eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.244+0000 7fe23d0ac700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2380fbe40 0x7fe238112960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe234009fd0 tx=0x7fe23400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.244+0000 7fe22dffb700 1 -- 172.21.15.2:0/2542728411 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe234004e90 con 0x7fe2380fbe40 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.244+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2381153f0 con 0x7fe2380fbe40 2024-02-16T22:21:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.244+0000 7fe22dffb700 1 -- 172.21.15.2:0/2542728411 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe23400f040 con 0x7fe2380fbe40 2024-02-16T22:21:26.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.245+0000 7fe22dffb700 1 -- 172.21.15.2:0/2542728411 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe23400ca50 con 0x7fe2380fbe40 2024-02-16T22:21:26.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.245+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2381157f0 con 0x7fe2380fbe40 2024-02-16T22:21:26.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.246+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23804bed0 con 0x7fe2380fbe40 2024-02-16T22:21:26.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.247+0000 7fe22dffb700 1 -- 172.21.15.2:0/2542728411 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe234007cc0 con 0x7fe2380fbe40 2024-02-16T22:21:26.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.248+0000 7fe22dffb700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe2240690a0 0x7fe22406b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:26.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.248+0000 7fe22dffb700 1 -- 172.21.15.2:0/2542728411 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe234086ad0 con 0x7fe2380fbe40 2024-02-16T22:21:26.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.248+0000 7fe23c8ab700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe2240690a0 0x7fe22406b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:26.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.249+0000 7fe23c8ab700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe2240690a0 0x7fe22406b550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe228013850 tx=0x7fe228009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:26.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.255+0000 7fe22dffb700 1 -- 172.21.15.2:0/2542728411 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe234052db0 con 0x7fe2380fbe40 2024-02-16T22:21:26.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:25 smithi103 ceph-mon[36259]: pgmap v252: 65 pgs: 65 active+clean; 61 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 455 op/s 2024-02-16T22:21:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.462+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe238115a30 con 0x7fe2240690a0 2024-02-16T22:21:26.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.480+0000 7fe22dffb700 1 -- 172.21.15.2:0/2542728411 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3344 (secure 0 0 0) 0x7fe238115a30 con 0x7fe2240690a0 2024-02-16T22:21:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-16T22:21:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.smithi002 smithi002 *:9093,9094 running (6m) 2m ago 7m 25.0M - 0.25.0 c8568f914cd2 2b7780bcc981 2024-02-16T22:21:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi002 smithi002 running (8m) 2m ago 8m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 c1d597f23cdd 2024-02-16T22:21:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:crash.smithi103 smithi103 running (7m) 2m ago 7m 6660k - 17.2.7-513-gb88fae66 870a0944ccb8 db2a59e09a51 2024-02-16T22:21:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.smithi002 smithi002 *:3000 running (6m) 2m ago 7m 87.3M - 9.4.7 954c08fa6188 14099b699363 2024-02-16T22:21:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi002.hhufhl smithi002 running (2m) 2m ago 2m 15.3M - 17.2.7-513-gb88fae66 870a0944ccb8 12d8ab360b57 2024-02-16T22:21:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi002.llkqse smithi002 running (2m) 2m ago 2m 10.5M - 17.2.7-513-gb88fae66 870a0944ccb8 8e266571411c 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi103.epsgnr smithi103 running (2m) 2m ago 2m 14.5M - 17.2.7-513-gb88fae66 870a0944ccb8 d43d0b5d182c 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:mds.cephfs.smithi103.kbizjo smithi103 running (2m) 2m ago 2m 14.6M - 17.2.7-513-gb88fae66 870a0944ccb8 86d4a2e35372 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi002.resmjv smithi002 *:9283 running (9m) 2m ago 9m 480M - 17.2.7-513-gb88fae66 870a0944ccb8 f9b132a0aa69 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.smithi103.ydydbo smithi103 *:8443,9283 running (7m) 2m ago 6m 433M - 17.2.7-513-gb88fae66 870a0944ccb8 463897d4efc5 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi002 smithi002 running (9m) 2m ago 9m 45.2M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 cc6b0b4bb587 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:mon.smithi103 smithi103 running (6m) 2m ago 6m 35.6M 2048M 17.2.7-513-gb88fae66 870a0944ccb8 3d198bee2a92 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi002 smithi002 *:9100 running (8m) 2m ago 8m 15.9M - 1.5.0 0da6a335fe13 e6146a57c445 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.smithi103 smithi103 *:9100 running (7m) 2m ago 7m 17.0M - 1.5.0 0da6a335fe13 0b21b215b785 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (6m) 2m ago 6m 48.7M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 a3f19d068baf 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (5m) 2m ago 5m 46.1M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 58697cab97b0 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (5m) 2m ago 5m 47.1M 4289M 17.2.7-513-gb88fae66 870a0944ccb8 634798c2dbad 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi103 running (4m) 2m ago 4m 43.9M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 a5332c698672 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi103 running (4m) 2m ago 4m 42.5M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 fa2a12f4b710 2024-02-16T22:21:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi103 running (4m) 2m ago 4m 42.1M 5312M 17.2.7-513-gb88fae66 870a0944ccb8 175f4eefd1f6 2024-02-16T22:21:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.smithi002 smithi002 *:9095 running (6m) 2m ago 7m 55.1M - 2.43.0 a07b618ecd1d d03f44725127 2024-02-16T22:21:26.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.483+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe2240690a0 msgr2=0x7fe22406b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:26.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.483+0000 7fe23f310700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe2240690a0 0x7fe22406b550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe228013850 tx=0x7fe228009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:26.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.483+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2380fbe40 msgr2=0x7fe238112960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:26.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.483+0000 7fe23f310700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2380fbe40 0x7fe238112960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe234009fd0 tx=0x7fe23400b040 comp rx=0 tx=0).stop 2024-02-16T22:21:26.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.483+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 shutdown_connections 2024-02-16T22:21:26.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.483+0000 7fe23f310700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7fe2240690a0 0x7fe22406b550 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.483+0000 7fe23f310700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2380fbe40 0x7fe238112960 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.483+0000 7fe23f310700 1 --2- 172.21.15.2:0/2542728411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2380fcfb0 0x7fe238114eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.483+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 >> 172.21.15.2:0/2542728411 conn(0x7fe2380f7500 msgr2=0x7fe238100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:26.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.484+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 shutdown_connections 2024-02-16T22:21:26.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.484+0000 7fe23f310700 1 -- 172.21.15.2:0/2542728411 wait complete. 2024-02-16T22:21:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.649+0000 7f0cfac25700 1 -- 172.21.15.2:0/2262267953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf40fda80 msgr2=0x7f0cf40ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.649+0000 7f0cfac25700 1 --2- 172.21.15.2:0/2262267953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf40fda80 0x7f0cf40ffe60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8004940 tx=0x7f0ce8009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.650+0000 7f0cfac25700 1 -- 172.21.15.2:0/2262267953 shutdown_connections 2024-02-16T22:21:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.650+0000 7f0cfac25700 1 --2- 172.21.15.2:0/2262267953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cf41003a0 0x7f0cf4102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.650+0000 7f0cfac25700 1 --2- 172.21.15.2:0/2262267953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf40fda80 0x7f0cf40ffe60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.650+0000 7f0cfac25700 1 -- 172.21.15.2:0/2262267953 >> 172.21.15.2:0/2262267953 conn(0x7f0cf40f74e0 msgr2=0x7f0cf40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.650+0000 7f0cfac25700 1 -- 172.21.15.2:0/2262267953 shutdown_connections 2024-02-16T22:21:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.650+0000 7f0cfac25700 1 -- 172.21.15.2:0/2262267953 wait complete. 2024-02-16T22:21:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.651+0000 7f0cfac25700 1 Processor -- start 2024-02-16T22:21:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.651+0000 7f0cfac25700 1 -- start start 2024-02-16T22:21:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.651+0000 7f0cfac25700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cf40fda80 0x7f0cf410fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.651+0000 7f0cfac25700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf41003a0 0x7f0cf410fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.651+0000 7f0cfac25700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf41105f0 con 0x7f0cf40fda80 2024-02-16T22:21:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.651+0000 7f0cfac25700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4110e90 con 0x7f0cf41003a0 2024-02-16T22:21:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.652+0000 7f0cf3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf41003a0 0x7f0cf410fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.652+0000 7f0cf3fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf41003a0 0x7f0cf410fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.2:34562/0 (socket says 172.21.15.2:34562) 2024-02-16T22:21:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.652+0000 7f0cf3fff700 1 -- 172.21.15.2:0/2162199585 learned_addr learned my addr 172.21.15.2:0/2162199585 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:21:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.652+0000 7f0cf89c1700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cf40fda80 0x7f0cf410fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.652+0000 7f0cf3fff700 1 -- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cf40fda80 msgr2=0x7f0cf410fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.652+0000 7f0cf3fff700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cf40fda80 0x7f0cf410fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.653+0000 7f0cf3fff700 1 -- 172.21.15.2:0/2162199585 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8005040 con 0x7f0cf41003a0 2024-02-16T22:21:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.653+0000 7f0cf3fff700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf41003a0 0x7f0cf410fff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000eb40 tx=0x7f0ce000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.653+0000 7f0cf17fa700 1 -- 172.21.15.2:0/2162199585 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce0007cc0 con 0x7f0cf41003a0 2024-02-16T22:21:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.653+0000 7f0cf17fa700 1 -- 172.21.15.2:0/2162199585 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ce000f040 con 0x7f0cf41003a0 2024-02-16T22:21:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.653+0000 7f0cf17fa700 1 -- 172.21.15.2:0/2162199585 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce000ca10 con 0x7f0cf41003a0 2024-02-16T22:21:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.653+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf4111110 con 0x7f0cf41003a0 2024-02-16T22:21:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.653+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf4111650 con 0x7f0cf41003a0 2024-02-16T22:21:26.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.654+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf404bed0 con 0x7f0cf41003a0 2024-02-16T22:21:26.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.656+0000 7f0cf17fa700 1 -- 172.21.15.2:0/2162199585 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f0ce000cb70 con 0x7f0cf41003a0 2024-02-16T22:21:26.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.657+0000 7f0cf17fa700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0ce40692b0 0x7f0ce406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:26.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.657+0000 7f0cf17fa700 1 -- 172.21.15.2:0/2162199585 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0ce00871e0 con 0x7f0cf41003a0 2024-02-16T22:21:26.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.657+0000 7f0cf89c1700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0ce40692b0 0x7f0ce406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:26.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.657+0000 7f0cf89c1700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0ce40692b0 0x7f0ce406b760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80136a0 tx=0x7f0ce8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:26.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:26.662+0000 7f0cf17fa700 1 -- 172.21.15.2:0/2162199585 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0ce0056ff0 con 0x7f0cf41003a0 2024-02-16T22:21:27.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.111+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0cf4105a10 con 0x7f0cf41003a0 2024-02-16T22:21:27.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.112+0000 7f0cf17fa700 1 -- 172.21.15.2:0/2162199585 <== mon.1 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f0ce0056b80 con 0x7f0cf41003a0 2024-02-16T22:21:27.114 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-16T22:21:27.114 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-02-16T22:21:27.114 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 2 2024-02-16T22:21:27.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:21:27.114 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-02-16T22:21:27.114 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 2 2024-02-16T22:21:27.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:21:27.114 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-02-16T22:21:27.114 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 6 2024-02-16T22:21:27.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:21:27.115 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": { 2024-02-16T22:21:27.115 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 4 2024-02-16T22:21:27.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-02-16T22:21:27.115 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-02-16T22:21:27.115 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.7-513-gb88fae66 (b88fae66e6ad896087939a6bde358f026d85f561) quincy (stable)": 14 2024-02-16T22:21:27.115 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-02-16T22:21:27.115 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-16T22:21:27.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0ce40692b0 msgr2=0x7f0ce406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:27.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0ce40692b0 0x7f0ce406b760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80136a0 tx=0x7f0ce8009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:27.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf41003a0 msgr2=0x7f0cf410fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:27.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf41003a0 0x7f0cf410fff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000eb40 tx=0x7f0ce000b040 comp rx=0 tx=0).stop 2024-02-16T22:21:27.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 shutdown_connections 2024-02-16T22:21:27.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f0ce40692b0 0x7f0ce406b760 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cf40fda80 0x7f0cf410fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 --2- 172.21.15.2:0/2162199585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cf41003a0 0x7f0cf410fff0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 >> 172.21.15.2:0/2162199585 conn(0x7f0cf40f74e0 msgr2=0x7f0cf40fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:27.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 shutdown_connections 2024-02-16T22:21:27.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.115+0000 7f0cfac25700 1 -- 172.21.15.2:0/2162199585 wait complete. 2024-02-16T22:21:27.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:26 smithi002 ceph-mon[29154]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:21:27.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.270+0000 7f972f255700 1 -- 172.21.15.2:0/960131886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 msgr2=0x7f97280fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:27.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.270+0000 7f972f255700 1 --2- 172.21.15.2:0/960131886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 0x7f97280fd320 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f971c004940 tx=0x7f971c009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:27.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.271+0000 7f972f255700 1 -- 172.21.15.2:0/960131886 shutdown_connections 2024-02-16T22:21:27.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.271+0000 7f972f255700 1 --2- 172.21.15.2:0/960131886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97280fe0e0 0x7f97280fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.271+0000 7f972f255700 1 --2- 172.21.15.2:0/960131886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 0x7f97280fd320 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.271+0000 7f972f255700 1 -- 172.21.15.2:0/960131886 >> 172.21.15.2:0/960131886 conn(0x7f97280f8620 msgr2=0x7f97280faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:27.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.271+0000 7f972f255700 1 -- 172.21.15.2:0/960131886 shutdown_connections 2024-02-16T22:21:27.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.271+0000 7f972f255700 1 -- 172.21.15.2:0/960131886 wait complete. 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.271+0000 7f972f255700 1 Processor -- start 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.271+0000 7f972f255700 1 -- start start 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.272+0000 7f972f255700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 0x7f9728071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.272+0000 7f972f255700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97280fe0e0 0x7f97280702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.272+0000 7f972f255700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97280708e0 con 0x7f97280fcef0 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.272+0000 7f972cff1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 0x7f9728071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.272+0000 7f972cff1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 0x7f9728071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39452/0 (socket says 172.21.15.2:39452) 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.272+0000 7f972cff1700 1 -- 172.21.15.2:0/1630147039 learned_addr learned my addr 172.21.15.2:0/1630147039 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.272+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9728070a50 con 0x7f97280fe0e0 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.272+0000 7f9727fff700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97280fe0e0 0x7f97280702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.272+0000 7f972cff1700 1 -- 172.21.15.2:0/1630147039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97280fe0e0 msgr2=0x7f97280702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.273+0000 7f972cff1700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97280fe0e0 0x7f97280702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.273+0000 7f972cff1700 1 -- 172.21.15.2:0/1630147039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c005040 con 0x7f97280fcef0 2024-02-16T22:21:27.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.273+0000 7f9727fff700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97280fe0e0 0x7f97280702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-16T22:21:27.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.273+0000 7f972cff1700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 0x7f9728071e60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f971c000c00 tx=0x7f971c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:27.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.273+0000 7f97257fa700 1 -- 172.21.15.2:0/1630147039 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f971c00e070 con 0x7f97280fcef0 2024-02-16T22:21:27.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.273+0000 7f97257fa700 1 -- 172.21.15.2:0/1630147039 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f971c01c400 con 0x7f97280fcef0 2024-02-16T22:21:27.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.273+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9728070cd0 con 0x7f97280fcef0 2024-02-16T22:21:27.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.273+0000 7f97257fa700 1 -- 172.21.15.2:0/1630147039 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f971c00b670 con 0x7f97280fcef0 2024-02-16T22:21:27.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.273+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97280710f0 con 0x7f97280fcef0 2024-02-16T22:21:27.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.275+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970c0052e0 con 0x7f97280fcef0 2024-02-16T22:21:27.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.275+0000 7f97257fa700 1 -- 172.21.15.2:0/1630147039 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f971c013af0 con 0x7f97280fcef0 2024-02-16T22:21:27.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.275+0000 7f97257fa700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f97180692a0 0x7f971806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:27.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.275+0000 7f97257fa700 1 -- 172.21.15.2:0/1630147039 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f971c088840 con 0x7f97280fcef0 2024-02-16T22:21:27.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.276+0000 7f9727fff700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f97180692a0 0x7f971806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:27.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.276+0000 7f9727fff700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f97180692a0 0x7f971806b750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9714009fd0 tx=0x7f971400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:27.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.284+0000 7f97257fa700 1 -- 172.21.15.2:0/1630147039 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f971c058590 con 0x7f97280fcef0 2024-02-16T22:21:27.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:26 smithi103 ceph-mon[36259]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:21:27.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.694+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f970c005da0 con 0x7f97280fcef0 2024-02-16T22:21:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.695+0000 7f97257fa700 1 -- 172.21.15.2:0/1630147039 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1830 (secure 0 0 0) 0x7f971c058120 con 0x7f97280fcef0 2024-02-16T22:21:27.700 INFO:teuthology.orchestra.run.smithi002.stdout:e16 2024-02-16T22:21:27.700 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-16T22:21:27.700 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 16 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-02-16T22:18:52.996722+0000 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2024-02-16T22:19:10.960547+0000 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 60 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2024-02-16T22:21:27.701 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 46 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 1 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:in 0 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24263} 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3] 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data enabled 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 1 2024-02-16T22:21:27.702 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.kbizjo{0:24263} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.103:6824/419104453,v1:172.21.15.103:6825/419104453] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:21:27.703 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.hhufhl{0:14486} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.2:6828/971745533,v1:172.21.15.2:6829/971745533] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:21:27.703 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:21:27.703 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:21:27.703 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2024-02-16T22:21:27.703 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-02-16T22:21:27.703 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi002.llkqse{-1:14496} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.2:6826/3769723051,v1:172.21.15.2:6827/3769723051] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:21:27.703 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.cephfs.smithi103.epsgnr{-1:24271} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.103:6826/1310923282,v1:172.21.15.103:6827/1310923282] compat {c=[1],r=[1],i=[7ff]}] 2024-02-16T22:21:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.698+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f97180692a0 msgr2=0x7f971806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.698+0000 7f972f255700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f97180692a0 0x7f971806b750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9714009fd0 tx=0x7f971400b040 comp rx=0 tx=0).stop 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.698+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 msgr2=0x7f9728071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.698+0000 7f972f255700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 0x7f9728071e60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f971c000c00 tx=0x7f971c009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.698+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 shutdown_connections 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.698+0000 7f972f255700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f97180692a0 0x7f971806b750 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.698+0000 7f972f255700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97280fcef0 0x7f9728071e60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.698+0000 7f972f255700 1 --2- 172.21.15.2:0/1630147039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97280fe0e0 0x7f97280702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.698+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 >> 172.21.15.2:0/1630147039 conn(0x7f97280f8620 msgr2=0x7f97281013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.699+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 shutdown_connections 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.699+0000 7f972f255700 1 -- 172.21.15.2:0/1630147039 wait complete. 2024-02-16T22:21:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 16 2024-02-16T22:21:27.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.825+0000 7f295f840700 1 -- 172.21.15.2:0/134109080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2958066c70 msgr2=0x7f2958101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:27.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.825+0000 7f295f840700 1 --2- 172.21.15.2:0/134109080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2958066c70 0x7f2958101f00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2954004940 tx=0x7f2954009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:27.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.826+0000 7f295f840700 1 -- 172.21.15.2:0/134109080 shutdown_connections 2024-02-16T22:21:27.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.826+0000 7f295f840700 1 --2- 172.21.15.2:0/134109080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2958102440 0x7f2958104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.826+0000 7f295f840700 1 --2- 172.21.15.2:0/134109080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2958066c70 0x7f2958101f00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.826+0000 7f295f840700 1 -- 172.21.15.2:0/134109080 >> 172.21.15.2:0/134109080 conn(0x7f29580f7510 msgr2=0x7f29580f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:27.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.826+0000 7f295f840700 1 -- 172.21.15.2:0/134109080 shutdown_connections 2024-02-16T22:21:27.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.826+0000 7f295f840700 1 -- 172.21.15.2:0/134109080 wait complete. 2024-02-16T22:21:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.826+0000 7f295f840700 1 Processor -- start 2024-02-16T22:21:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295f840700 1 -- start start 2024-02-16T22:21:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295f840700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2958066c70 0x7f295810fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295f840700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2958102440 0x7f29581100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295f840700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29581106e0 con 0x7f2958102440 2024-02-16T22:21:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295f840700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2958110f80 con 0x7f2958066c70 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295cddb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2958102440 0x7f29581100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295d5dc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2958066c70 0x7f295810fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295d5dc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2958066c70 0x7f295810fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.2:34584/0 (socket says 172.21.15.2:34584) 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295d5dc700 1 -- 172.21.15.2:0/1819077931 learned_addr learned my addr 172.21.15.2:0/1819077931 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295d5dc700 1 -- 172.21.15.2:0/1819077931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2958102440 msgr2=0x7f29581100e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295d5dc700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2958102440 0x7f29581100e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.827+0000 7f295d5dc700 1 -- 172.21.15.2:0/1819077931 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2954005040 con 0x7f2958066c70 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.828+0000 7f295d5dc700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2958066c70 0x7f295810fba0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f29540093b0 tx=0x7f2954009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.828+0000 7f294dffb700 1 -- 172.21.15.2:0/1819077931 <== mon.1 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f295400e040 con 0x7f2958066c70 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.828+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2958111200 con 0x7f2958066c70 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.828+0000 7f294dffb700 1 -- 172.21.15.2:0/1819077931 <== mon.1 v2:172.21.15.103:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f295400bcc0 con 0x7f2958066c70 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.829+0000 7f294dffb700 1 -- 172.21.15.2:0/1819077931 <== mon.1 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f295401c630 con 0x7f2958066c70 2024-02-16T22:21:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.829+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2958111680 con 0x7f2958066c70 2024-02-16T22:21:27.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.830+0000 7f294dffb700 1 -- 172.21.15.2:0/1819077931 <== mon.1 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f2954013420 con 0x7f2958066c70 2024-02-16T22:21:27.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.830+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295804bed0 con 0x7f2958066c70 2024-02-16T22:21:27.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.830+0000 7f294dffb700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29440692a0 0x7f294406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:27.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.830+0000 7f294dffb700 1 -- 172.21.15.2:0/1819077931 <== mon.1 v2:172.21.15.103:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2954088430 con 0x7f2958066c70 2024-02-16T22:21:27.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.831+0000 7f295cddb700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29440692a0 0x7f294406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:27.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.831+0000 7f295cddb700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29440692a0 0x7f294406b750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f29580669b0 tx=0x7f294800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:27.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:27.835+0000 7f294dffb700 1 -- 172.21.15.2:0/1819077931 <== mon.1 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2954058240 con 0x7f2958066c70 2024-02-16T22:21:27.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:27 smithi002 ceph-mon[29154]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:21:27.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:27 smithi002 ceph-mon[29154]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:21:27.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:27 smithi002 ceph-mon[29154]: pgmap v253: 65 pgs: 65 active+clean; 61 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 455 op/s 2024-02-16T22:21:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:27 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/2162199585' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-16T22:21:27.976 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:27 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/1630147039' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:21:28.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.129+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 --> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2958111e00 con 0x7f29440692a0 2024-02-16T22:21:28.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.130+0000 7f294dffb700 1 -- 172.21.15.2:0/1819077931 <== mgr.14212 v2:172.21.15.2:6800/1176790314 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f2958111e00 con 0x7f29440692a0 2024-02-16T22:21:28.132 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-02-16T22:21:28.132 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-16T22:21:28.132 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-02-16T22:21:28.132 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-16T22:21:28.132 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-02-16T22:21:28.132 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "0/21 daemons upgraded", 2024-02-16T22:21:28.132 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 image on host smithi103", 2024-02-16T22:21:28.132 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-02-16T22:21:28.132 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29440692a0 msgr2=0x7f294406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29440692a0 0x7f294406b750 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f29580669b0 tx=0x7f294800b040 comp rx=0 tx=0).stop 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2958066c70 msgr2=0x7f295810fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2958066c70 0x7f295810fba0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f29540093b0 tx=0x7f2954009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 shutdown_connections 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f29440692a0 0x7f294406b750 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2958066c70 0x7f295810fba0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 --2- 172.21.15.2:0/1819077931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2958102440 0x7f29581100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 >> 172.21.15.2:0/1819077931 conn(0x7f29580f7510 msgr2=0x7f29580f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 shutdown_connections 2024-02-16T22:21:28.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.133+0000 7f295f840700 1 -- 172.21.15.2:0/1819077931 wait complete. 2024-02-16T22:21:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.269+0000 7f82ca677700 1 -- 172.21.15.2:0/4021175442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82c41003a0 msgr2=0x7f82c4102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.269+0000 7f82ca677700 1 --2- 172.21.15.2:0/4021175442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82c41003a0 0x7f82c4102780 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f82b8004940 tx=0x7f82b8009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.269+0000 7f82ca677700 1 -- 172.21.15.2:0/4021175442 shutdown_connections 2024-02-16T22:21:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.269+0000 7f82ca677700 1 --2- 172.21.15.2:0/4021175442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82c41003a0 0x7f82c4102780 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.269+0000 7f82ca677700 1 --2- 172.21.15.2:0/4021175442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82c40fda80 0x7f82c40ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.270+0000 7f82ca677700 1 -- 172.21.15.2:0/4021175442 >> 172.21.15.2:0/4021175442 conn(0x7f82c40f74e0 msgr2=0x7f82c40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.270+0000 7f82ca677700 1 -- 172.21.15.2:0/4021175442 shutdown_connections 2024-02-16T22:21:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.270+0000 7f82ca677700 1 -- 172.21.15.2:0/4021175442 wait complete. 2024-02-16T22:21:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.270+0000 7f82ca677700 1 Processor -- start 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.271+0000 7f82ca677700 1 -- start start 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.271+0000 7f82ca677700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82c40fda80 0x7f82c40fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.271+0000 7f82ca677700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82c41003a0 0x7f82c40fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.271+0000 7f82ca677700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c41115c0 con 0x7f82c40fda80 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.271+0000 7f82ca677700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c4111730 con 0x7f82c41003a0 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.271+0000 7f82c37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82c41003a0 0x7f82c40fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.271+0000 7f82c3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82c40fda80 0x7f82c40fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.271+0000 7f82c37fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82c41003a0 0x7f82c40fb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.2:34614/0 (socket says 172.21.15.2:34614) 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.271+0000 7f82c37fe700 1 -- 172.21.15.2:0/3739589353 learned_addr learned my addr 172.21.15.2:0/3739589353 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.272+0000 7f82c3fff700 1 -- 172.21.15.2:0/3739589353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82c41003a0 msgr2=0x7f82c40fb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.272+0000 7f82c3fff700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82c41003a0 0x7f82c40fb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.272+0000 7f82c3fff700 1 -- 172.21.15.2:0/3739589353 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8005040 con 0x7f82c40fda80 2024-02-16T22:21:28.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.272+0000 7f82c37fe700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82c41003a0 0x7f82c40fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-16T22:21:28.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.272+0000 7f82c3fff700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82c40fda80 0x7f82c40fd510 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009fd0 tx=0x7f82b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:28.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.272+0000 7f82c0ff9700 1 -- 172.21.15.2:0/3739589353 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b4004e90 con 0x7f82c40fda80 2024-02-16T22:21:28.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.272+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c40fbf90 con 0x7f82c40fda80 2024-02-16T22:21:28.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.273+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c40fc4d0 con 0x7f82c40fda80 2024-02-16T22:21:28.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.273+0000 7f82c0ff9700 1 -- 172.21.15.2:0/3739589353 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82b4004e90 con 0x7f82c40fda80 2024-02-16T22:21:28.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.273+0000 7f82c0ff9700 1 -- 172.21.15.2:0/3739589353 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b400c7d0 con 0x7f82c40fda80 2024-02-16T22:21:28.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.274+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b00052e0 con 0x7f82c40fda80 2024-02-16T22:21:28.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.274+0000 7f82c0ff9700 1 -- 172.21.15.2:0/3739589353 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f82b4007cc0 con 0x7f82c40fda80 2024-02-16T22:21:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.275+0000 7f82c0ff9700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f82ac069360 0x7f82ac06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-16T22:21:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.275+0000 7f82c0ff9700 1 -- 172.21.15.2:0/3739589353 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f82b40869c0 con 0x7f82c40fda80 2024-02-16T22:21:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.275+0000 7f82c37fe700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f82ac069360 0x7f82ac06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-16T22:21:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.276+0000 7f82c37fe700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f82ac069360 0x7f82ac06b810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f82b800b0d0 tx=0x7f82b8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-16T22:21:28.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.280+0000 7f82c0ff9700 1 -- 172.21.15.2:0/3739589353 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f82b404dbd0 con 0x7f82c40fda80 2024-02-16T22:21:28.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:27 smithi103 ceph-mon[36259]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:21:28.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:27 smithi103 ceph-mon[36259]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:21:28.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:27 smithi103 ceph-mon[36259]: pgmap v253: 65 pgs: 65 active+clean; 61 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 455 op/s 2024-02-16T22:21:28.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:27 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/2162199585' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-16T22:21:28.326 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:27 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/1630147039' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-16T22:21:28.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.580+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f82b0005590 con 0x7f82c40fda80 2024-02-16T22:21:28.582 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-16T22:21:28.582 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-16T22:21:28.582 INFO:teuthology.orchestra.run.smithi002.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-16T22:21:28.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.580+0000 7f82c0ff9700 1 -- 172.21.15.2:0/3739589353 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f82b4056760 con 0x7f82c40fda80 2024-02-16T22:21:28.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.582+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f82ac069360 msgr2=0x7f82ac06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:28.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.582+0000 7f82ca677700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f82ac069360 0x7f82ac06b810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f82b800b0d0 tx=0x7f82b8009e30 comp rx=0 tx=0).stop 2024-02-16T22:21:28.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.582+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82c40fda80 msgr2=0x7f82c40fd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-16T22:21:28.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.582+0000 7f82ca677700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82c40fda80 0x7f82c40fd510 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009fd0 tx=0x7f82b400b040 comp rx=0 tx=0).stop 2024-02-16T22:21:28.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.582+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 shutdown_connections 2024-02-16T22:21:28.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.582+0000 7f82ca677700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:6800/1176790314,v1:172.21.15.2:6801/1176790314] conn(0x7f82ac069360 0x7f82ac06b810 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:28.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.582+0000 7f82ca677700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82c40fda80 0x7f82c40fd510 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009fd0 tx=0x7f82b400b040 comp rx=0 tx=0).stop 2024-02-16T22:21:28.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.582+0000 7f82ca677700 1 --2- 172.21.15.2:0/3739589353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82c41003a0 0x7f82c40fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-16T22:21:28.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.582+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 >> 172.21.15.2:0/3739589353 conn(0x7f82c40f74e0 msgr2=0x7f82c40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-16T22:21:28.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.583+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 shutdown_connections 2024-02-16T22:21:28.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:21:28.583+0000 7f82ca677700 1 -- 172.21.15.2:0/3739589353 wait complete. 2024-02-16T22:21:28.592 INFO:tasks.workunit.client.1.smithi103.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-16T22:21:28.592 INFO:tasks.workunit.client.1.smithi103.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-16T22:21:28.592 INFO:tasks.workunit.client.1.smithi103.stderr:+ make 2024-02-16T22:21:28.643 INFO:tasks.workunit.client.1.smithi103.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-16T22:21:29.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:28 smithi002 ceph-mon[29154]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:21:29.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:28 smithi002 ceph-mon[29154]: from='client.? 172.21.15.2:0/3739589353' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-16T22:21:29.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:28 smithi002 ceph-mon[29154]: pgmap v254: 65 pgs: 65 active+clean; 61 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 454 op/s 2024-02-16T22:21:29.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:28 smithi103 ceph-mon[36259]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-16T22:21:29.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:28 smithi103 ceph-mon[36259]: from='client.? 172.21.15.2:0/3739589353' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-16T22:21:29.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:28 smithi103 ceph-mon[36259]: pgmap v254: 65 pgs: 65 active+clean; 61 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 454 op/s 2024-02-16T22:21:32.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:31 smithi002 ceph-mon[29154]: pgmap v255: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.8 MiB/s rd, 6.7 MiB/s wr, 591 op/s 2024-02-16T22:21:32.325 INFO:journalctl@ceph.mon.smithi103.smithi103.stdout:Feb 16 22:21:31 smithi103 ceph-mon[36259]: pgmap v255: 65 pgs: 65 active+clean; 78 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.8 MiB/s rd, 6.7 MiB/s wr, 591 op/s 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to 'ed7ab725e1a43615009dfb74a42bf33d80081bc8'. 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2024-02-16T22:21:32.396 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-02-16T22:21:32.397 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2024-02-16T22:21:32.397 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-02-16T22:21:32.397 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2024-02-16T22:21:32.397 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-02-16T22:21:32.397 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-16T22:21:32.397 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-02-16T22:21:32.397 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at ed7ab725e1a qa: test fuse/kclient for mds upgrade seq 2024-02-16T22:21:32.401 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-16T22:21:32.465 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-16T22:21:32.467 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-16T22:21:32.467 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-16T22:21:33.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:21:32 smithi002 ceph-mon[29154]: pgmap v256: 65 pgs: 65 active+clean; 80 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.4 MiB/s wr, 309 op/s 2024-02-16T22:21:33.250 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-16T22:21:33.314 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-16T22:21:33.353 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-16T22:21:33.355 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-16T22:21:33.355 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-16T22:21:33.393 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-16T22:21:33.396 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:21:33.396 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-16T22:21:33.451 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-16T22:21:33.453 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-16T22:21:33.453 DEBUG:teuthology.orchestra.run.smithi002:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed7ab725e1a43615009dfb74a42bf33d80081bc8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-16T22:21:55.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:55.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:55.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:55.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:55.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:55.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:55.614+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:55.614+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:55.614+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:56.550+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:56.550+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:56.550+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:56.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:56.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:56.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:57.612 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:57.587+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:57.612 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:57.587+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:57.613 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:57.587+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:57.613 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:21:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:21:57.346+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:21:57.613 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:21:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:21:57.346+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:21:57.613 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:21:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:21:57.346+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:21:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:57.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:57.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:57.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:58.646 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:58.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:58.646 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:58.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:58.647 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:58.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:58.647 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:21:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:21:58.371+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:21:58.647 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:21:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:21:58.371+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:21:58.647 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:21:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:21:58.371+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:21:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:58.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:58.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:58.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:59.631 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:21:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:21:59.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:21:59.632 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:21:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:21:59.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:21:59.632 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:21:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:21:59.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:21:59.929 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:59.628+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:59.929 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:59.628+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:59.929 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:21:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:21:59.628+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:21:59.930 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:59.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:59.930 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:59.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:21:59.930 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:21:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:21:59.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:00.712 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:00.601+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:00.712 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:00.601+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:00.712 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:00.601+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:00.712 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:00.373+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:00.712 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:00.373+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:00.712 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:00.373+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:00.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:00.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:00.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:01.653 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:01.326+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:01.653 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:01.326+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:01.653 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:01.326+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:01.732+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:01.732+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:01.732+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:01.650+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:01.650+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:01.650+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:02.638 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:02.342+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:02.638 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:02.342+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:02.638 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:02.342+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:02.892 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:02.706+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:02.892 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:02.706+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:02.892 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:02.706+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:02.892 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:02.636+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:02.892 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:02.636+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:02.892 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:02.636+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:03.685 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:03.371+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:03.685 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:03.371+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:03.685 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:03.371+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:03.686 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:03.588+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:03.686 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:03.588+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:03.686 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:03.588+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:03.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:03.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:03.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:03.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:03.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:03.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:04.703 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:04.377+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:04.704 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:04.377+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:04.704 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:04.377+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:04.704 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:04.568+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:04.704 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:04.568+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:04.704 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:04.568+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:04.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:04.701+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:04.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:04.701+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:04.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:04.701+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:05.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:05.335+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:05.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:05.335+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:05.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:05.335+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:05.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:05.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:05.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:06.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:05.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:06.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:05.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:06.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:05.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:06.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:06.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:06.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:06.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:06.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:06.727 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:06.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:06.727 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:06.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:07.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:06.732+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:07.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:06.732+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:07.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:06.732+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:07.702 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:07.501+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:07.702 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:07.501+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:07.702 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:07.501+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:07.702 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:07.302+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:07.702 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:07.302+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:07.702 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:07.302+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:07.699+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:07.699+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:07.699+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:08.336+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:08.336+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:08.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:08.336+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:08.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:08.468+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:08.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:08.468+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:08.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:08.468+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:09.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:08.748+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:09.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:08.748+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:09.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:08.748+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:09.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:09.307+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:09.307+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:09.307+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:09.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:09.459+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:09.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:09.459+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:09.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:09.459+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:10.046 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:09.775+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:10.046 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:09.775+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:10.046 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:09.775+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:10.307+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:10.307+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:10.307+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:10.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:10.414+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:10.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:10.414+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:10.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:10.414+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:11.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:10.756+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:11.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:10.756+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:11.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:10.756+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:11.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:11.290+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:11.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:11.290+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:11.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:11.290+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:11.369+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:11.369+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:11.369+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:12.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:11.770+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:12.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:11.770+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:12.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:11.770+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:12.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:12.397+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:12.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:12.397+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:12.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:12.397+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:12.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:12.250+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:12.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:12.250+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:12.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:12.250+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:13.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:12.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:13.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:12.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:13.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:12.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:13.512 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:13.255+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:13.512 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:13.255+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:13.512 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:13.255+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:13.512 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:13.395+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:13.512 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:13.395+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:13.512 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:13.395+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:13.756+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:13.756+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:13.756+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:14.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:14.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:14.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:14.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:14.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:14.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:14.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:14.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:14.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:14.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:14.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:15.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:15.253+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:15.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:15.253+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:15.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:15.253+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:15.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:15.417+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:15.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:15.417+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:15.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:15.417+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:16.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:15.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:16.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:15.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:16.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:15.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:16.385+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:16.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:16.385+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:16.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:16.385+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:16.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:16.265+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:16.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:16.265+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:16.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:16.265+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:17.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:16.734+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:17.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:16.734+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:17.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:16.734+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:17.715 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:17.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:17.716 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:17.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:17.716 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:17.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:17.716 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:17.286+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:17.716 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:17.286+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:17.716 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:17.286+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:17.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:17.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:17.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:18.722 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:18.332+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:18.722 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:18.332+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:18.722 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:18.332+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:18.722 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:18.308+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:18.722 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:18.308+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:18.722 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:18.308+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:18.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:18.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:18.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:18.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:18.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:18.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:19.681 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:19.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:19.681 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:19.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:19.681 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:19.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:19.682 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:19.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:19.682 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:19.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:19.682 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:19.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:19.932 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:19.679+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:19.932 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:19.679+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:19.932 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:19.679+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:20.676 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:20.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:20.676 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:20.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:20.676 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:20.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:20.676 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:20.293+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:20.677 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:20.293+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:20.677 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:20.293+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:20.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:20.674+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:20.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:20.674+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:20.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:20.674+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:21.656 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:21.317+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:21.656 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:21.317+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:21.656 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:21.317+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:21.656 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:21.273+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:21.656 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:21.273+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:21.656 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:21.273+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:21.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:21.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:21.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:22.641 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:22.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:22.641 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:22.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:22.641 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:22.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:22.642 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:22.247+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:22.642 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:22.247+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:22.642 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:22.247+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:22.639+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:22.639+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:22.639+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:23.513 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:23.318+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:23.513 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:23.318+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:23.513 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:23.318+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:23.514 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:23.240+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:23.514 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:23.240+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:23.514 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:23.240+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:23.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:23.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:23.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:23.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:23.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:23.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:24.612 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:24.235+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:24.613 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:24.235+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:24.613 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:24.235+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:24.613 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:24.331+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:24.613 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:24.331+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:24.613 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:24.331+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:24.921 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:24.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:24.922 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:24.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:24.922 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:24.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:25.584 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:25.328+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:25.584 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:25.328+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:25.584 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:25.328+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:25.584 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:25.260+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:25.584 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:25.260+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:25.584 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:25.260+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:25.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:25.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:25.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:26.626 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:26.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:26.626 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:26.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:26.626 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:26.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:26.627 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:26.292+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:26.627 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:26.292+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:26.627 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:26.292+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:26.955 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:26.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:26.956 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:26.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:26.956 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:26.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:27.623 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:27.337+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:27.623 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:27.337+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:27.623 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:27.337+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:27.623 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:27.262+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:27.623 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:27.262+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:27.623 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:27.262+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:27.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:27.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:27.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:28.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:28.335+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:28.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:28.335+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:28.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:28.335+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:28.606 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:28.263+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:28.606 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:28.263+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:28.606 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:28.263+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:28.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:28.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:28.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:29.644 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:29.291+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:29.644 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:29.291+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:29.644 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:29.291+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:29.645 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:29.297+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:29.645 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:29.297+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:29.645 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:29.297+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:29.932 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:29.642+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:29.932 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:29.642+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:29.932 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:29.642+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:30.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:30.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:30.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:30.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:30.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:30.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:30.635 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:30.301+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:30.635 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:30.301+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:30.635 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:30.301+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:30.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:30.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:30.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:30.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:30.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:31.612 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:31.367+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:31.613 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:31.367+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:31.613 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:31.367+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:31.613 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:31.285+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:31.613 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:31.285+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:31.613 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:31.285+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:31.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:31.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:31.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:32.610 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:32.356+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:32.611 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:32.356+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:32.611 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:32.356+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:32.611 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:32.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:32.611 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:32.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:32.611 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:32.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:32.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:32.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:32.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:33.619 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:33.381+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:33.620 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:33.381+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:33.620 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:33.381+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:33.620 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:33.304+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:33.620 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:33.304+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:33.620 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:33.304+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:33.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:33.617+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:33.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:33.617+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:33.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:33.617+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:34.669 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:34.352+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:34.669 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:34.352+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:34.669 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:34.352+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:34.669 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:34.331+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:34.670 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:34.331+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:34.670 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:34.331+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:34.935 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:34.667+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:34.935 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:34.667+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:34.935 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:34.667+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:35.634 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:35.355+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:35.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:35.355+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:35.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:35.355+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:35.635 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:35.349+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:35.635 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:35.349+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:35.635 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:35.349+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:35.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:35.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:35.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:36.671 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:36.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:36.671 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:36.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:36.671 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:36.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:36.672 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:36.308+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:36.672 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:36.308+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:36.672 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:36.308+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:36.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:36.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:36.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:37.660 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:37.412+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:37.660 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:37.412+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:37.660 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:37.412+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:37.660 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:37.265+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:37.660 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:37.265+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:37.661 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:37.265+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:37.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:37.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:37.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:38.677 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:38.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:38.677 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:38.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:38.677 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:38.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:38.677 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:38.245+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:38.677 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:38.246+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:38.677 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:38.246+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:38.675+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:38.675+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:38.675+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:39.399+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:39.399+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:39.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:39.399+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:39.222+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:39.222+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:39.222+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:39.927 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:39.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:39.927 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:39.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:39.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:39.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:40.213+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:40.213+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:40.213+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:40.407+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:40.407+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:40.407+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:40.687+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:40.687+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:40.687+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:41.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:41.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:41.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:41.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:41.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:41.219+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:41.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:41.219+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:41.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:41.219+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:41.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:41.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:41.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:42.337+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:42.337+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:42.337+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:42.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:42.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:42.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:42.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:42.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:42.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:42.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:42.642+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:42.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:42.642+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:42.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:42.642+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:43.343+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:43.343+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:43.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:43.343+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:43.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:43.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:43.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:43.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:43.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:43.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:44.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:44.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:44.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:44.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:44.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:44.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:44.933 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:44.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:44.933 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:44.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:44.933 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:44.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:45.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:45.165+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:45.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:45.165+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:45.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:45.165+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:45.679 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:45.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:45.679 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:45.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:45.679 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:45.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:45.677+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:45.677+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:45.677+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:46.143+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:46.143+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:46.143+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:46.281+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:46.281+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:46.281+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:46.697+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:46.697+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:46.697+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:47.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:47.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:47.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:47.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:47.295+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:47.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:47.295+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:47.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:47.295+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:47.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:47.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:47.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:48.333 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:48.125+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:48.334 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:48.125+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:48.334 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:48.125+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:48.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:48.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:48.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:48.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:48.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:48.331+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:49.124 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:48.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:49.124 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:48.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:49.124 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:48.742+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:49.312+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:49.312+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:49.312+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:49.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:49.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:49.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:49.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:49.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:49.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:50.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:50.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:50.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:50.158+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:50.158+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:50.158+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:50.703+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:50.703+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:50.703+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:51.246+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:51.246+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:51.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:51.246+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:51.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:51.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:51.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:51.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:51.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:51.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:52.181 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:51.725+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:52.182 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:51.725+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:52.182 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:51.725+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:52.293+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:52.293+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:52.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:52.293+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:52.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:52.179+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:52.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:52.179+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:52.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:52.179+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:53.204 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:52.740+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:53.204 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:52.740+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:53.204 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:52.740+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:53.315+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:53.315+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:53.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:53.315+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:53.201+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:53.201+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:53.201+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:53.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:53.730+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:53.730+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:53.730+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:54.309+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:54.309+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:54.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:54.309+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:54.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:54.194+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:54.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:54.194+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:54.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:54.194+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:54.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:54.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:54.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:54.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:54.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:54.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:55.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:55.321+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:55.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:55.321+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:55.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:55.321+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:55.223+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:55.223+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:55.223+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:55.662+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:55.662+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:55.662+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:56.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:56.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:56.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:56.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:56.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:56.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:56.198+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:56.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:56.198+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:56.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:56.198+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:56.696+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:56.696+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:56.696+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:57.345+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:57.345+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:57.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:57.345+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:57.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:57.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:57.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:57.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:57.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:57.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:58.318+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:58.318+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:58.318+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:58.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:58.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:58.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:58.708+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:58.708+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:58.708+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:22:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:59.343+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:59.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:59.343+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:59.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:22:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:22:59.343+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:22:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:59.116+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:59.116+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:22:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:22:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:22:59.116+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:00.047 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:59.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:00.047 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:59.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:00.047 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:22:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:22:59.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:00.363 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:00.099+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:00.364 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:00.099+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:00.364 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:00.099+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:00.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:00.373+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:00.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:00.373+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:00.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:00.373+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:01.079 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:00.765+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:01.079 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:00.765+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:01.079 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:00.765+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:01.408 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:01.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:01.408 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:01.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:01.408 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:01.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:01.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:01.406+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:01.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:01.406+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:01.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:01.406+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:02.119 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:01.804+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:02.119 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:01.804+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:02.119 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:01.804+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:02.423 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:02.117+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:02.423 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:02.117+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:02.423 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:02.117+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:02.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:02.421+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:02.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:02.421+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:02.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:02.421+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:03.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:02.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:03.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:02.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:03.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:02.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:03.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:03.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:03.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:03.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:03.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:03.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:03.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:03.441+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:03.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:03.441+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:03.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:03.441+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:04.105 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:03.730+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:04.105 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:03.730+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:04.105 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:03.730+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:04.405 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:04.103+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:04.405 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:04.103+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:04.405 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:04.103+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:04.716 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:04.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:04.716 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:04.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:04.716 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:04.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:04.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:04.714+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:04.714+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:04.714+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:05.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:05.092+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:05.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:05.092+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:05.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:05.092+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:05.680 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:05.451+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:05.681 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:05.451+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:05.681 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:05.451+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:05.679+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:05.679+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:05.679+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:06.413 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:06.142+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:06.413 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:06.142+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:06.413 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:06.142+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:06.701 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:06.411+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:06.701 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:06.411+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:06.702 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:06.411+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:06.700+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:06.700+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:06.700+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:07.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:07.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:07.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:07.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:07.362+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:07.362+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:07.362+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:07.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:07.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:07.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:08.337+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:08.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:08.337+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:08.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:08.337+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:08.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:08.138+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:08.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:08.138+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:08.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:08.138+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:08.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:08.687+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:08.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:08.687+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:08.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:08.687+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:09.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:09.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:09.371+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:09.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:09.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:09.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:09.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:09.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:09.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:09.935 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:09.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:09.936 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:09.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:09.936 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:09.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:10.407+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:10.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:10.407+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:10.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:10.407+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:10.215+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:10.215+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:10.215+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:10.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:10.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:10.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:11.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:11.199+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:11.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:11.199+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:11.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:11.199+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:11.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:11.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:11.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:11.450+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:11.450+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:11.450+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:12.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:12.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:12.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:12.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:12.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:12.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:12.480+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:12.480+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:12.480+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:13.473 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:13.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:13.473 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:13.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:13.473 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:13.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:13.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:13.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:13.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:13.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:13.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:13.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:13.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:13.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:13.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:13.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:13.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:14.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:14.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:14.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:14.214+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:14.214+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:14.214+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:14.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:14.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:14.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:14.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:14.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:14.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:15.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:15.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:15.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:15.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:15.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:15.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:15.601 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:15.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:15.601 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:15.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:15.601 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:15.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:15.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:15.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:15.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:15.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:15.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:16.433 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:16.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:16.433 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:16.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:16.433 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:16.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:16.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:16.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:16.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:16.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:16.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:16.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:17.383 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:17.097+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:17.384 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:17.097+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:17.384 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:17.097+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:17.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:17.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:17.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:17.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:17.381+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:17.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:17.381+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:17.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:17.381+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:18.431 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:18.058+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:18.431 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:18.058+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:18.431 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:18.058+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:18.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:18.562+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:18.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:18.562+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:18.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:18.562+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:18.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:18.429+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:18.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:18.429+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:18.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:18.429+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:19.059+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:19.059+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:19.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:19.059+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:19.926 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:19.475+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:19.927 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:19.475+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:19.927 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:19.475+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:19.927 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:19.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:19.927 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:19.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:19.927 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:19.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:20.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:20.021+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:20.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:20.021+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:20.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:20.021+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:20.496+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:20.496+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:20.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:20.496+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:20.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:20.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:20.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:21.464 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:21.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:21.464 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:21.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:21.464 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:21.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:21.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:21.462+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:21.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:21.462+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:21.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:21.462+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:21.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:21.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:21.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:21.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:21.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:21.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:22.030+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:22.030+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:22.030+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:22.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:22.501+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:22.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:22.501+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:22.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:22.501+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:22.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:22.501+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:22.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:22.501+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:22.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:22.501+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:23.042+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:23.042+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:23.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:23.042+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:23.549+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:23.549+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:23.549+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:23.514+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:23.514+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:23.514+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:24.076+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:24.076+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:24.076+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:24.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:24.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:24.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:24.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:24.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:24.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:24.937 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:24.539+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:24.937 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:24.539+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:24.937 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:24.539+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:25.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:25.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:25.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:25.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:25.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:25.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:25.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:25.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:25.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:25.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:25.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:25.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:25.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:26.014+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:26.014+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:26.014+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:26.522+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:26.522+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:26.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:26.522+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:26.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:26.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:26.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:27.063+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:27.063+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:27.063+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:27.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:27.508+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:27.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:27.508+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:27.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:27.508+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:27.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:27.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:27.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:28.468 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:28.068+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:28.468 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:28.068+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:28.468 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:28.068+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:28.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:28.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:28.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:28.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:28.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:28.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:28.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:28.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:28.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:29.427 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:29.096+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:29.427 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:29.096+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:29.427 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:29.096+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:29.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:29.425+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:29.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:29.425+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:29.425+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:29.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:29.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:29.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:29.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:29.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:29.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:30.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:30.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:30.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:30.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:30.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:30.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:30.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:30.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:30.581+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:30.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:30.581+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:30.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:30.581+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:31.457 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:31.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:31.457 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:31.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:31.458 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:31.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:31.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:31.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:31.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:31.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:31.592+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:31.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:31.592+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:31.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:31.592+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:32.457 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:32.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:32.458 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:32.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:32.458 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:32.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:32.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:32.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:32.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:32.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:32.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:32.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:33.441 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:33.040+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:33.441 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:33.040+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:33.441 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:33.040+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:33.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:33.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:33.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:33.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:33.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:33.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:33.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:33.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:33.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:34.446 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:33.994+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:34.446 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:33.994+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:34.447 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:33.994+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:34.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:34.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:34.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:34.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:34.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:34.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:34.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:35.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:35.016+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:35.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:35.016+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:35.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:35.016+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:35.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:35.517+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:35.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:35.517+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:35.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:35.517+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:35.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:35.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:35.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:35.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:35.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:35.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:36.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:36.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:36.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:36.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:36.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:36.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:36.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:36.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:37.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:37.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:37.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:37.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:37.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:37.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:37.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:37.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:37.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:37.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:38.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:38.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:38.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:38.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:38.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:38.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:38.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:38.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:38.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:39.473 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:39.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:39.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:39.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:39.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:39.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:39.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:39.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:39.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:39.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:39.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:39.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:39.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:39.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:39.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:40.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:23:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:23:39.926+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:23:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:40.027+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:40.027+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:40.027+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:40.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:40.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:40.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:40.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:40.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:40.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:40.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:40.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:40.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:40.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:40.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:40.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:41.412 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:41.041+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:41.412 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:41.041+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:41.412 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:41.041+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:41.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:41.482+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:41.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:41.482+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:41.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:41.482+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:41.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:41.410+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:41.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:41.410+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:41.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:41.410+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:42.394 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:42.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:42.394 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:42.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:42.394 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:42.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:42.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:42.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:42.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:42.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:42.390+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:42.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:42.390+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:42.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:42.390+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:43.371 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:43.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:43.371 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:43.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:43.371 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:43.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:43.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:43.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:43.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:43.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:43.441+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:43.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:43.441+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:43.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:43.441+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:44.411 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:44.033+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:44.412 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:44.033+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:44.412 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:44.033+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:44.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:44.518+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:44.518+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:44.518+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:44.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:44.409+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:44.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:44.409+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:44.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:44.409+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:45.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:23:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:23:44.927+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:23:45.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:45.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:45.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:45.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:45.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:45.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:45.569 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:45.454+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:45.569 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:45.454+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:45.569 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:45.454+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:45.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:45.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:45.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:46.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:46.042+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:46.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:46.042+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:46.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:46.042+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:46.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:46.456+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:46.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:46.456+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:46.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:46.456+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:46.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:46.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:46.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:46.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:46.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:46.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:46.996+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:46.996+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:46.996+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:47.953 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:47.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:47.953 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:47.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:47.953 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:47.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:47.953 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:47.508+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:47.953 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:47.508+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:47.953 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:47.508+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:47.951+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:47.951+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:47.951+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:48.947 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:48.496+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:48.948 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:48.496+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:48.948 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:48.496+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:48.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:48.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:48.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:48.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:48.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:48.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:48.945+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:48.945+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:48.945+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:49.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:49.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:49.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:49.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:49.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:49.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:49.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:49.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:49.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:49.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:49.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:49.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:50.053 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:23:49.928+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 31 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:23:50.054 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:49.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:50.054 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:49.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:50.054 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:49.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:50.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:50.441+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:50.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:50.441+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:50.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:50.441+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:50.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:50.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:50.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:50.890+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:50.890+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:50.890+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:51.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:51.415+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:51.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:51.415+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:51.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:51.415+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:51.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:51.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:51.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:51.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:51.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:51.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:51.909+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:51.909+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:51.909+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:52.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:52.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:52.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:52.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:52.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:52.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:52.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:52.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:52.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:52.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:52.959+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:52.959+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:52.959+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:53.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:53.432+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:53.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:53.432+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:53.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:53.432+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:53.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:53.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:53.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:53.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:53.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:53.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:53.918+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:53.918+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:53.918+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:54.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:54.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:54.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:54.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:54.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:54.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:54.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:54.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:54.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:23:54.929+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 53 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:23:55.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:54.967+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:55.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:54.967+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:55.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:54.967+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:55.498 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:55.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:55.498 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:55.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:55.498 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:55.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:55.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:55.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:55.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:56.433 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:55.980+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:56.433 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:55.980+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:56.433 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:55.980+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:56.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:56.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:56.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:56.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:56.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:56.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:56.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:56.461+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:56.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:56.461+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:56.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:56.461+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:56.964+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:56.964+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:56.964+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:57.459+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:57.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:57.459+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:57.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:57.459+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:57.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:57.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:57.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:57.969+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:57.969+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:57.969+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:58.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:58.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:58.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:58.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:58.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:58.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:58.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:58.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:58.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:58.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:58.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:58.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:59.405 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:58.997+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:59.405 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:58.997+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:59.405 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:23:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:23:58.997+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:23:59.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:59.427+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:59.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:59.427+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:59.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:23:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:23:59.427+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:23:59.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:59.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:59.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:59.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:23:59.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:23:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:23:59.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:00.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:23:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:23:59.929+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 81 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:00.011+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:00.011+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:00.011+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:00.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:00.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:00.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:00.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:00.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:00.403+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:00.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:00.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:00.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:01.406 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:01.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:01.406 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:01.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:01.406 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:01.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:01.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:01.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:01.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:01.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:01.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:01.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:01.405+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:01.405+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:01.405+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:02.040+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:02.040+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:02.040+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:02.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:02.442+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:02.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:02.442+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:02.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:02.442+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:02.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:02.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:02.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:03.414 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:03.069+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:03.414 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:03.069+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:03.414 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:03.069+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:03.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:03.467+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:03.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:03.467+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:03.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:03.467+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:03.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:03.412+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:03.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:03.412+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:03.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:03.412+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:04.413 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:04.054+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:04.413 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:04.054+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:04.414 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:04.054+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:04.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:04.491+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:04.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:04.491+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:04.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:04.491+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:04.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:04.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:04.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:04.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:04.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:04.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:05.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:04.930+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 111 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:05.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:05.054+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:05.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:05.054+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:05.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:05.054+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:05.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:05.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:05.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:05.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:05.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:05.363+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:05.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:05.363+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:05.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:05.363+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:06.384 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:06.088+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:06.384 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:06.088+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:06.384 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:06.088+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:06.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:06.474+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:06.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:06.474+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:06.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:06.474+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:06.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:06.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:06.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:06.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:06.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:06.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:07.400 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:07.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:07.401 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:07.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:07.401 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:07.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:07.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:07.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:07.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:07.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:07.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:07.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:07.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:07.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:07.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:07.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:07.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:07.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:08.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:08.053+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:08.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:08.053+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:08.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:08.053+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:08.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:08.500+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:08.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:08.500+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:08.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:08.500+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:08.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:08.381+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:08.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:08.381+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:08.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:08.381+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:09.414 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:09.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:09.415 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:09.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:09.415 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:09.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:09.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:09.488+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:09.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:09.488+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:09.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:09.488+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:09.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:09.413+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:09.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:09.413+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:09.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:09.413+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:09.930+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 140 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:10.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:10.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:10.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:10.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:10.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:10.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:10.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:10.457+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:10.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:10.457+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:10.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:10.457+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:10.446+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:10.446+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:10.446+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:11.416 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:11.015+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:11.416 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:11.015+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:11.416 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:11.015+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:11.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:11.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:11.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:11.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:11.477+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:11.477+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:11.477+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:12.421 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:12.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:12.421 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:12.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:12.421 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:12.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:12.419+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:12.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:12.419+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:12.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:12.419+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:12.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:12.499+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:12.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:12.499+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:12.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:12.499+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:13.376 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:13.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:13.376 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:13.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:13.376 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:13.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:13.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:13.374+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:13.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:13.374+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:13.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:13.374+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:13.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:13.543+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:13.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:13.543+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:13.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:13.543+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:14.330 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:14.074+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:14.330 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:14.074+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:14.331 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:14.074+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:14.544+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:14.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:14.544+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:14.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:14.544+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:14.329+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:14.329+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:14.329+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:15.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:14.931+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 167 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:15.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:15.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:15.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:15.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:15.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:15.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:15.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:15.535+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:15.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:15.535+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:15.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:15.535+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:15.290+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:15.290+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:15.290+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:16.070+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:16.070+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:16.070+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:16.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:16.255+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:16.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:16.255+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:16.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:16.255+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:16.512+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:16.512+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:16.512+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:17.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:17.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:17.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:17.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:17.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:17.232+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:17.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:17.232+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:17.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:17.232+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:17.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:17.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:17.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:17.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:17.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:17.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:18.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:18.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:18.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:18.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:18.275+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:18.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:18.275+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:18.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:18.275+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:18.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:18.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:18.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:19.116+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:19.116+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:19.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:19.116+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:19.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:19.262+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:19.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:19.262+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:19.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:19.262+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:19.934 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:19.520+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:19.934 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:19.520+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:19.934 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:19.520+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:20.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:19.931+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 199 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:20.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:20.093+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:20.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:20.093+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:20.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:20.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:20.531 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:20.269+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:20.531 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:20.269+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:20.531 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:20.269+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:20.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:20.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:20.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:21.084+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:21.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:21.084+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:21.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:21.084+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:21.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:21.302+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:21.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:21.302+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:21.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:21.302+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:21.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:21.558+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:21.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:21.558+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:21.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:21.558+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:22.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:22.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:22.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:22.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:22.315+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:22.315+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:22.315+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:22.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:22.534+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:22.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:22.534+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:22.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:22.534+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:23.088+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:23.088+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:23.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:23.088+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:23.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:23.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:23.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:23.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:23.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:23.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:23.522+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:23.522+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:23.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:23.522+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:24.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:24.287+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:24.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:24.287+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:24.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:24.287+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:24.095+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:24.095+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:24.095+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:24.934 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:24.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:24.934 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:24.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:24.935 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:24.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:25.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:24.932+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 232 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:25.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:25.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:25.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:25.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:25.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:25.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:25.554 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:25.272+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:25.554 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:25.272+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:25.554 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:25.272+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:25.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:25.552+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:25.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:25.552+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:25.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:25.552+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:26.152+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:26.152+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:26.152+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:26.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:26.302+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:26.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:26.302+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:26.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:26.302+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:26.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:26.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:26.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:27.110+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:27.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:27.110+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:27.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:27.110+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:27.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:27.295+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:27.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:27.295+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:27.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:27.295+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:27.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:27.536+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:27.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:27.536+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:27.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:27.536+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:28.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:28.138+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:28.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:28.138+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:28.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:28.138+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:28.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:28.309+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:28.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:28.309+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:28.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:28.309+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:28.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:28.496+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:28.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:28.496+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:28.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:28.496+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:29.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:29.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:29.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:29.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:29.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:29.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:29.351+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:29.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:29.351+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:29.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:29.351+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:29.935 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:29.494+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:29.935 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:29.494+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:29.935 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:29.494+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:30.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:29.933+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 259 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:30.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:30.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:30.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:30.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:30.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:30.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:30.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:30.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:30.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:30.367+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:30.367+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:30.367+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:31.391 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:31.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:31.391 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:31.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:31.391 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:31.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:31.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:31.516+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:31.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:31.516+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:31.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:31.516+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:31.389+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:31.389+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:31.389+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:32.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:32.175+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:32.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:32.175+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:32.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:32.175+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:32.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:32.409+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:32.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:32.409+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:32.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:32.409+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:32.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:32.556+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:32.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:32.556+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:32.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:32.556+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:33.153+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:33.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:33.153+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:33.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:33.153+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:33.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:33.394+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:33.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:33.394+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:33.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:33.394+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:33.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:33.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:33.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:33.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:34.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:34.147+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:34.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:34.147+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:34.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:34.147+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:34.598+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:34.598+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:34.598+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:34.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:34.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:34.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:35.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:34.934+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 286 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:35.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:35.157+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:35.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:35.157+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:35.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:35.157+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:35.562 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:35.413+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:35.562 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:35.413+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:35.562 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:35.413+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:35.560+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:35.560+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:35.560+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:36.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:36.427+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:36.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:36.427+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:36.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:36.427+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:36.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:36.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:36.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:36.570+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:36.570+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:36.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:36.570+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:37.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:37.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:37.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:37.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:37.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:37.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:37.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:37.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:37.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:37.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:37.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:37.584+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:37.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:37.584+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:37.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:37.584+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:38.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:38.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:38.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:38.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:38.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:38.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:38.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:38.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:38.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:38.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:38.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:38.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:38.605+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:38.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:38.605+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:38.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:38.605+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:39.120+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:39.120+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:39.120+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:39.936 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:39.566+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:39.936 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:39.566+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:39.936 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:39.566+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:39.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:39.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:39.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:39.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:39.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:39.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:40.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:39.934+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 313 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:40.143+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:40.143+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:40.143+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:40.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:40.431+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:40.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:40.431+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:40.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:40.431+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:40.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:40.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:40.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:40.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:40.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:40.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:41.451 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:41.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:41.451 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:41.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:41.451 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:41.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:41.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:41.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:41.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:41.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:41.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:41.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:41.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:41.561+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:41.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:41.561+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:41.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:41.561+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:42.413 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:42.111+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:42.413 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:42.111+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:42.413 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:42.111+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:42.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:42.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:42.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:42.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:42.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:42.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:42.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:42.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:42.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:42.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:42.583+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:43.430 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:43.120+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:43.430 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:43.120+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:43.430 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:43.120+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:43.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:43.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:43.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:43.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:43.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:43.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:43.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:43.428+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:43.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:43.428+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:43.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:43.428+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:44.449 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:44.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:44.449 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:44.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:44.449 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:44.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:44.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:44.447+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:44.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:44.447+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:44.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:44.447+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:44.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:44.541+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:44.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:44.541+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:44.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:44.541+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:45.061 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:44.935+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 342 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:45.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:45.157+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:45.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:45.157+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:45.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:45.157+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:45.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:45.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:45.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:45.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:45.514+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:45.514+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:45.514+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:46.124+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:46.124+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:46.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:46.124+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:46.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:46.547+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:46.547+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:46.547+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:46.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:46.534+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:46.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:46.534+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:46.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:46.534+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:47.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:47.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:47.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:47.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:47.507+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:47.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:47.507+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:47.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:47.507+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:47.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:47.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:47.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:48.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:48.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:48.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:48.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:48.531+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:48.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:48.531+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:48.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:48.531+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:48.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:48.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:48.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:49.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:49.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:49.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:49.937 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:49.480+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:49.938 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:49.480+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:49.938 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:49.480+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:49.938 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:49.559+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:49.938 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:49.560+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:49.938 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:49.560+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:50.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:49.935+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 369 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:50.157+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:50.157+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:50.157+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:50.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:50.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:50.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:50.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:50.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:50.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:50.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:50.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:50.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:51.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:51.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:51.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:51.540 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:51.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:51.541 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:51.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:51.541 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:51.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:51.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:51.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:51.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:52.383+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:52.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:52.383+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:52.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:52.383+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:52.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:52.205+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:52.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:52.205+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:52.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:52.205+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:52.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:52.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:52.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:53.522 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:53.342+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:53.522 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:53.342+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:53.522 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:53.342+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:53.523 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:53.244+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:53.523 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:53.244+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:53.523 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:53.244+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:53.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:53.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:53.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:53.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:53.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:53.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:54.304 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:54.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:54.304 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:54.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:54.304 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:54.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:54.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:54.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:54.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:54.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:54.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:54.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:54.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:54.302+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:54.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:54.302+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:54.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:54.302+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:54.936+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 396 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:24:55.485 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:55.324+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:55.486 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:55.324+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:55.486 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:55.324+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:55.486 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:55.238+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:55.486 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:55.238+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:55.486 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:55.238+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:55.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:55.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:55.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:56.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:56.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:56.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:56.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:56.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:56.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:56.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:56.296+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:56.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:56.296+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:56.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:56.296+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:56.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:56.280+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:56.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:56.280+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:56.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:56.280+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:57.311+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:57.311+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:57.311+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:57.270+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:57.270+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:57.270+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:57.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:57.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:57.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:58.284+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:58.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:58.284+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:58.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:58.284+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:58.222+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:58.222+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:58.222+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:58.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:58.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:58.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:59.288+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:59.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:59.288+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:59.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:24:59.288+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:24:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:59.187+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:59.187+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:24:59.187+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:24:59.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:59.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:59.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:59.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:24:59.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:24:59.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:00.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:24:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:24:59.937+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 423 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:00.598 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:00.228+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:00.598 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:00.228+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:00.598 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:00.228+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:00.599 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:00.245+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:00.599 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:00.245+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:00.599 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:00.245+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:00.597+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:00.597+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:00.597+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:01.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:01.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:01.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:01.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:01.223+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:01.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:01.223+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:01.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:01.223+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:01.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:01.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:01.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:02.210+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:02.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:02.210+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:02.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:02.210+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:02.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:02.231+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:02.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:02.231+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:02.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:02.231+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:02.587+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:02.587+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:02.587+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:03.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:03.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:03.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:03.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:03.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:03.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:03.254+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:03.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:03.254+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:03.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:03.254+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:03.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:03.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:03.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:03.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:03.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:03.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:04.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:04.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:04.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:04.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:04.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:04.287+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:04.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:04.287+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:04.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:04.287+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:04.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:04.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:04.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:04.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:04.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:04.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:05.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:04.938+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 455 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:05.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:05.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:05.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:05.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:05.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:05.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:05.566 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:05.312+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:05.566 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:05.312+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:05.566 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:05.312+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:05.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:05.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:05.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:06.359 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:06.086+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:06.359 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:06.086+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:06.359 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:06.086+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:06.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:06.357+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:06.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:06.357+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:06.357+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:06.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:06.520+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:06.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:06.520+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:06.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:06.520+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:07.129+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:07.129+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:07.129+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:07.363+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:07.363+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:07.363+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:07.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:07.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:07.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:08.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:08.099+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:08.377 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:08.099+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:08.377 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:08.099+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:08.374+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:08.374+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:08.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:08.374+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:08.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:08.452+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:08.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:08.452+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:08.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:08.452+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:09.387 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:09.073+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:09.388 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:09.073+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:09.388 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:09.073+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:09.413+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:09.413+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:09.413+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:09.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:09.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:09.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:09.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:09.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:09.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:09.937+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 488 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:10.075+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:10.075+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:10.075+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:10.440+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:10.440+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:10.440+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:10.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:10.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:10.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:11.435 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:11.066+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:11.435 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:11.066+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:11.435 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:11.066+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:11.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:11.432+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:11.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:11.432+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:11.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:11.432+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:11.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:11.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:11.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:12.419 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:12.084+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:12.420 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:12.084+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:12.420 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:12.084+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:12.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:12.416+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:12.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:12.416+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:12.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:12.416+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:12.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:12.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:12.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:12.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:12.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:12.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:13.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:13.097+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:13.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:13.097+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:13.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:13.097+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:13.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:13.402+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:13.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:13.402+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:13.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:13.402+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:13.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:13.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:13.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:14.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:14.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:14.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:14.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:14.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:14.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:14.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:14.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:14.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:14.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:14.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:14.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:14.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:15.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:14.938+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 515 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:15.154+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:15.154+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:15.154+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:15.334+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:15.334+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:15.334+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:15.500+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:15.500+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:15.500+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:16.382 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:16.122+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:16.382 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:16.122+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:16.382 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:16.122+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:16.380+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:16.380+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:16.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:16.380+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:16.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:16.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:16.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:17.421 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:17.103+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:17.421 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:17.103+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:17.421 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:17.103+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:17.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:17.419+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:17.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:17.419+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:17.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:17.419+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:17.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:17.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:17.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:18.418 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:18.058+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:18.418 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:18.058+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:18.418 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:18.058+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:18.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:18.416+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:18.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:18.416+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:18.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:18.416+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:18.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:18.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:18.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:19.451 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:19.019+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:19.452 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:19.019+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:19.452 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:19.019+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:19.500+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:19.500+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:19.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:19.500+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:19.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:19.450+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:19.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:19.450+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:19.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:19.450+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:20.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:19.939+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 542 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:20.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:20.014+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:20.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:20.014+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:20.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:20.014+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:20.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:20.438+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:20.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:20.438+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:20.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:20.438+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:20.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:20.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:20.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:21.410 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:21.033+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:21.410 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:21.033+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:21.411 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:21.033+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:21.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:21.408+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:21.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:21.408+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:21.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:21.408+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:21.675 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:21.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:21.675 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:21.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:21.675 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:21.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:22.368 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:22.050+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:22.368 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:22.050+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:22.368 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:22.050+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:22.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:22.366+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:22.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:22.366+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:22.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:22.366+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:22.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:22.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:22.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:23.410 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:23.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:23.410 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:23.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:23.410 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:23.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:23.408+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:23.408+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:23.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:23.408+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:23.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:23.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:23.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:24.460 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:24.125+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:24.460 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:24.125+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:24.460 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:24.125+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:24.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:24.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:24.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:24.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:24.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:24.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:24.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:25.067 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:24.940+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 571 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:25.457 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:25.108+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:25.458 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:25.108+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:25.458 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:25.108+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:25.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:25.456+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:25.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:25.456+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:25.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:25.456+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:25.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:25.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:25.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:25.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:25.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:25.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:26.425 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:26.064+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:26.426 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:26.064+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:26.426 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:26.064+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:26.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:26.424+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:26.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:26.424+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:26.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:26.424+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:26.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:26.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:26.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:26.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:26.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:26.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:27.394 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:27.108+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:27.394 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:27.108+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:27.394 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:27.108+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:27.635+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:27.635+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:27.635+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:27.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:27.392+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:27.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:27.392+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:27.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:27.392+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:28.429 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:28.059+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:28.429 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:28.059+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:28.429 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:28.059+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:28.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:28.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:28.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:28.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:28.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:28.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:28.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:28.640+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:28.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:28.640+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:28.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:28.640+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:29.403 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:29.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:29.403 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:29.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:29.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:29.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:29.682 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:29.401+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:29.682 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:29.401+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:29.683 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:29.401+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:29.942 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:29.680+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:29.943 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:29.680+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:29.943 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:29.680+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:30.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:29.940+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 598 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:30.018+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:30.018+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:30.018+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:30.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:30.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:30.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:30.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:30.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:30.450+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:30.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:30.450+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:30.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:30.450+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:31.432 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:31.033+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:31.432 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:31.033+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:31.432 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:31.033+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:31.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:31.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:31.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:31.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:31.430+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:31.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:31.430+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:31.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:31.430+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:32.420 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:32.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:32.420 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:32.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:32.420 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:32.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:32.670 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:32.667+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:32.670 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:32.667+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:32.670 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:32.667+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:32.670 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:32.418+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:32.670 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:32.418+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:32.670 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:32.418+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:33.464 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:33.111+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:33.464 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:33.111+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:33.464 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:33.111+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:33.462+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:33.462+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:33.462+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:33.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:33.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:33.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:34.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:34.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:34.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:34.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:34.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:34.724 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:34.418+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:34.724 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:34.418+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:34.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:34.418+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:34.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:34.941+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 625 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:34.722+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:34.722+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:34.722+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:35.380 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:35.130+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:35.380 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:35.130+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:35.380 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:35.130+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:35.678 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:35.376+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:35.678 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:35.376+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:35.678 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:35.376+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:35.676+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:35.676+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:35.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:35.676+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:36.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:36.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:36.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:36.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:36.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:36.367+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:36.367+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:36.367+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:36.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:36.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:36.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:37.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:37.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:37.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:37.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:37.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:37.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:37.350+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:37.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:37.350+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:37.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:37.350+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:37.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:37.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:37.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:38.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:38.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:38.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:38.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:38.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:38.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:38.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:38.389+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:38.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:38.389+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:38.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:38.389+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:38.721+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:38.721+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:38.721+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:39.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:39.203+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:39.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:39.203+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:39.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:39.203+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:39.384+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:39.384+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:39.384+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:39.966 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:39.942+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 652 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:39.966 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:39.705+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:39.966 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:39.705+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:39.966 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:39.705+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:40.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:40.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:40.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:40.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:40.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:40.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:40.662 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:40.410+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:40.662 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:40.410+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:40.663 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:40.410+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:40.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:40.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:40.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:41.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:41.128+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:41.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:41.128+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:41.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:41.128+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:41.674 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:41.415+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:41.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:41.415+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:41.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:41.415+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:41.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:41.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:41.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:42.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:42.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:42.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:42.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:42.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:42.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:42.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:42.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:42.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:42.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:42.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:42.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:42.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:43.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:43.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:43.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:43.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:43.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:43.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:43.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:43.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:43.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:43.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:43.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:44.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:44.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:44.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:44.362+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:44.362+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:44.362+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:44.945 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:44.676+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:44.945 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:44.676+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:44.945 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:44.676+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:45.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:44.943+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 681 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:45.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:45.136+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:45.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:45.136+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:45.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:45.136+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:45.658 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:45.317+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:45.658 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:45.317+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:45.658 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:45.317+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:45.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:45.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:45.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:46.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:46.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:46.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:46.096+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:46.096+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:46.096+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:46.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:46.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:46.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:47.101+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:47.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:47.101+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:47.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:47.101+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:47.319+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:47.319+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:47.319+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:48.128 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:47.726+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:48.129 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:47.726+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:48.129 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:47.726+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:48.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:48.126+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:48.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:48.126+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:48.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:48.126+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:48.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:48.297+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:48.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:48.297+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:48.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:48.297+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:48.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:48.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:48.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:49.106+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:49.106+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:49.106+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:49.276+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:49.276+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:49.276+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:49.957 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:49.943+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 708 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:49.958 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:49.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:49.958 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:49.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:49.958 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:49.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:50.088+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:50.088+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:50.088+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:50.663 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:50.272+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:50.663 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:50.272+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:50.663 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:50.272+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:50.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:50.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:50.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:51.311 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:51.046+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:51.311 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:51.046+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:51.312 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:51.046+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:51.636 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:51.309+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:51.637 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:51.309+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:51.637 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:51.309+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:51.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:51.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:51.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:52.309 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:52.055+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:52.309 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:52.055+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:52.309 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:52.055+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:52.632 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:52.306+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:52.632 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:52.306+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:52.632 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:52.306+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:52.630+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:52.630+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:52.630+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:53.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:53.351+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:53.351+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:53.351+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:53.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:53.101+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:53.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:53.101+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:53.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:53.101+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:53.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:53.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:53.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:53.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:54.321+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:54.321+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:54.321+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:54.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:54.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:54.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:54.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:54.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:54.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:54.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:54.640+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:54.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:54.640+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:54.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:54.640+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:54.943+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 735 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:25:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:55.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:55.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:55.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:55.603 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:55.276+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:55.603 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:55.276+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:55.603 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:55.276+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:55.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:55.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:55.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:56.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:56.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:56.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:56.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:56.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:56.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:56.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:56.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:56.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:56.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:56.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:56.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:56.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:56.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:57.113+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:57.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:57.113+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:57.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:57.113+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:57.236+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:57.236+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:57.236+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:57.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:57.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:57.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:58.133+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:58.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:58.133+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:58.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:58.133+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:58.248+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:58.248+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:58.248+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:58.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:58.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:58.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:59.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:59.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:59.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:25:59.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:25:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:59.236+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:59.236+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:25:59.236+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:25:59.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:59.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:59.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:59.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:25:59.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:25:59.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:00.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:25:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:25:59.944+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 762 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:00.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:00.123+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:00.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:00.123+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:00.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:00.123+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:00.635 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:00.280+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:00.635 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:00.280+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:00.635 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:00.280+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:00.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:00.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:00.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:01.109+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:01.109+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:01.109+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:01.311+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:01.311+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:01.311+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:01.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:01.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:01.628+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:02.109+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:02.109+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:02.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:02.109+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:02.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:02.329+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:02.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:02.329+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:02.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:02.329+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:02.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:02.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:02.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:03.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:03.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:03.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:03.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:03.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:03.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:03.359+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:03.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:03.359+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:03.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:03.359+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:03.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:03.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:03.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:04.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:04.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:04.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:04.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:04.381+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:04.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:04.381+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:04.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:04.381+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:04.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:04.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:04.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:04.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:04.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:04.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:05.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:04.945+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 791 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:05.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:05.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:05.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:05.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:05.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:05.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:05.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:05.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:05.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:05.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:05.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:05.386+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:05.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:05.386+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:05.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:05.386+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:06.164+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:06.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:06.164+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:06.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:06.164+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:06.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:06.356+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:06.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:06.356+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:06.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:06.356+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:06.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:06.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:06.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:07.180+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:07.180+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:07.180+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:07.407+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:07.407+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:07.407+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:07.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:07.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:07.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:08.378 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:08.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:08.378 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:08.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:08.378 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:08.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:08.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:08.375+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:08.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:08.375+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:08.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:08.375+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:08.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:08.578+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:08.578+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:08.579+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:09.400+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:09.400+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:09.400+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:09.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:09.153+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:09.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:09.153+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:09.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:09.153+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:09.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:09.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:09.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:09.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:09.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:09.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:09.946+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 817 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:10.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:10.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:10.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:10.425+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:10.425+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:10.425+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:10.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:10.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:10.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:11.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:11.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:11.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:11.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:11.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:11.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:11.378+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:11.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:11.378+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:11.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:11.378+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:11.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:11.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:11.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:12.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:12.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:12.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:12.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:12.356+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:12.356+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:12.356+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:12.632+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:12.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:12.632+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:12.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:12.632+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:13.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:13.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:13.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:13.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:13.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:13.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:13.377+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:13.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:13.377+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:13.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:13.377+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:13.899 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:13.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:13.899 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:13.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:13.899 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:13.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:14.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:14.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:14.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:14.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:14.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:14.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:14.368+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:14.368+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:14.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:14.368+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:14.559+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:14.559+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:14.559+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:15.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:14.947+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 845 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:15.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:15.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:15.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:15.604 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:15.380+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:15.604 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:15.380+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:15.604 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:15.380+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:15.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:15.602+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:15.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:15.602+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:15.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:15.602+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:16.135+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:16.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:16.135+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:16.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:16.135+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:16.342+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:16.342+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:16.342+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:16.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:16.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:16.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:16.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:16.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:16.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:17.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:17.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:17.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:17.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:17.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:17.296+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:17.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:17.296+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:17.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:17.296+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:17.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:17.601+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:17.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:17.601+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:17.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:17.601+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:18.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:18.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:18.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:18.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:18.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:18.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:18.287+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:18.287+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:18.287+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:18.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:18.596+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:18.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:18.596+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:18.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:18.596+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:19.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:19.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:19.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:19.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:19.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:19.334+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:19.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:19.334+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:19.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:19.334+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:19.949 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:19.547+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:19.949 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:19.547+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:19.949 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:19.547+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:20.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:19.947+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 871 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:20.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:20.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:20.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:20.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:20.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:20.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:20.375+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:20.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:20.375+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:20.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:20.375+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:20.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:20.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:20.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:21.419 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:21.113+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:21.419 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:21.113+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:21.419 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:21.113+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:21.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:21.417+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:21.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:21.417+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:21.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:21.417+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:21.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:21.547+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:21.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:21.547+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:21.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:21.547+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:22.444 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:22.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:22.444 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:22.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:22.444 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:22.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:22.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:22.443+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:22.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:22.443+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:22.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:22.443+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:22.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:22.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:22.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:23.456 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:23.073+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:23.456 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:23.073+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:23.457 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:23.073+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:23.455+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:23.455+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:23.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:23.455+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:23.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:23.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:23.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:24.429 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:24.026+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:24.429 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:24.026+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:24.429 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:24.026+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:24.428+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:24.428+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:24.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:24.428+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:24.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:24.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:24.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:25.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:24.948+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 906 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:25.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:25.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:25.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:25.501 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:25.455+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:25.502 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:25.455+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:25.502 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:25.455+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:25.500+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:25.500+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:25.500+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:26.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:26.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:26.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:26.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:26.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:26.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:26.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:26.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:26.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:26.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:26.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:26.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:27.055+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:27.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:27.055+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:27.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:27.055+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:27.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:27.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:27.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:27.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:27.511+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:27.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:27.511+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:27.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:27.511+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:28.455 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:28.029+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:28.455 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:28.029+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:28.455 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:28.029+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:28.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:28.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:28.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:28.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:28.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:28.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:28.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:28.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:28.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:29.425 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:28.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:29.425 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:28.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:29.425 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:28.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:29.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:29.540+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:29.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:29.540+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:29.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:29.540+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:29.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:29.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:29.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:30.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:29.949+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 947 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:29.987+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:29.987+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:29.987+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:30.538+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:30.538+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:30.538+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:30.469+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:30.469+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:30.469+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:31.438 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:31.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:31.439 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:31.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:31.439 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:31.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:31.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:31.555+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:31.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:31.555+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:31.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:31.555+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:31.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:31.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:31.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:32.331 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:31.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:32.331 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:31.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:32.331 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:31.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:32.606 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:32.439+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:32.606 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:32.439+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:32.606 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:32.439+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:32.605+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:32.605+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:32.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:32.605+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:33.391 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:33.028+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:33.391 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:33.028+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:33.392 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:33.028+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:33.390+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:33.390+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:33.390+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:33.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:33.585+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:33.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:33.585+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:33.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:33.585+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:34.375 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:34.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:34.375 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:34.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:34.375 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:34.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:34.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:34.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:34.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:34.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:34.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:34.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:34.374+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:34.374+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:34.374+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:35.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:34.950+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1025 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:35.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:35.060+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:35.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:35.060+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:35.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:35.060+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:35.664 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:35.361+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:35.664 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:35.361+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:35.664 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:35.361+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:35.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:35.661+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:35.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:35.662+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:35.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:35.662+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:36.335 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:36.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:36.336 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:36.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:36.336 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:36.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:36.684 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:36.334+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:36.684 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:36.334+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:36.684 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:36.334+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:36.969 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:36.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:36.969 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:36.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:36.969 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:36.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:37.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:36.966+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:37.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:36.966+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:37.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:36.966+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:37.708 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:37.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:37.708 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:37.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:37.708 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:37.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:37.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:37.947+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:37.947+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:37.947+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:37.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:37.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:37.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:37.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:37.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:37.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:38.691 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:38.325+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:38.691 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:38.325+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:38.691 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:38.325+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:38.688+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:38.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:38.688+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:38.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:38.688+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:38.910+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:38.910+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:38.910+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:39.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:39.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:39.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:39.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:39.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:39.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:39.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:39.949+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1101 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:39.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:39.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:39.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:39.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:39.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:39.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:39.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:39.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:39.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:39.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:39.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:39.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:40.687 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:40.365+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:40.687 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:40.365+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:40.687 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:40.365+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:40.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:40.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:40.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:40.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:40.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:40.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:40.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:40.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:40.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:41.640 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:41.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:41.640 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:41.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:41.640 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:41.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:41.921 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:41.724+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:41.921 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:41.724+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:41.921 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:41.724+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:42.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:41.918+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:42.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:41.918+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:42.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:41.918+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:42.692 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:42.420+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:42.692 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:42.420+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:42.693 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:42.420+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:42.932+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:42.932+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:42.932+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:42.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:42.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:42.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:42.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:42.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:42.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:43.721 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:43.427+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:43.722 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:43.427+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:43.722 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:43.427+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:43.920+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:43.920+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:43.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:43.920+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:43.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:43.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:43.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:43.719+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:43.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:43.719+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:44.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:44.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:44.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:44.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:44.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:44.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:44.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:44.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:44.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:45.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:44.950+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1181 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:45.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:44.897+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:45.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:44.897+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:45.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:44.897+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:45.702 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:45.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:45.703 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:45.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:45.703 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:45.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:45.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:45.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:45.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:45.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:45.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:45.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:45.700+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:45.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:45.700+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:45.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:45.700+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:46.904 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:46.742+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:46.905 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:46.742+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:46.905 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:46.742+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:46.905 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:46.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:46.905 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:46.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:46.905 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:46.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:46.902+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:46.902+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:46.902+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:47.786 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:47.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:47.786 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:47.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:47.786 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:47.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:47.912+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:47.912+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:47.912+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:47.783+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:47.783+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:47.783+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:48.756 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:48.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:48.756 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:48.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:48.756 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:48.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:48.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:48.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:49.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:48.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:48.754+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:48.754+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:48.754+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:49.719 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:49.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:49.719 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:49.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:49.719 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:49.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:49.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:49.952+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1257 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:49.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:49.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:49.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:49.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:49.897+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:49.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:49.897+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:49.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:49.897+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:50.701 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:50.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:50.701 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:50.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:50.701 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:50.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:50.699+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:50.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:50.699+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:50.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:50.699+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:50.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:50.899+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:50.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:50.899+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:50.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:50.899+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:51.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:51.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:51.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:51.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:51.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:51.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:51.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:51.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:51.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:51.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:51.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:51.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:51.857+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:51.857+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:51.857+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:52.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:52.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:52.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:52.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:52.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:52.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:52.727 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:52.430+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:52.727 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:52.430+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:52.727 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:52.430+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:52.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:52.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:52.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:53.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:53.605+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:53.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:53.605+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:53.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:53.605+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:53.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:53.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:53.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:53.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:53.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:53.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:53.815+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:54.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:53.815+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:54.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:53.815+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:54.861 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:54.633+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:54.861 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:54.633+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:54.861 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:54.633+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:54.861 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:54.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:54.861 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:54.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:54.861 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:54.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:54.953+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1335 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:26:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:54.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:54.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:54.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:55.872 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:55.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:55.872 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:55.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:55.872 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:55.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:55.873 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:55.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:55.873 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:55.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:55.873 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:55.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:55.870+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:55.870+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:55.870+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:56.862 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:56.691+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:56.863 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:56.691+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:56.863 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:56.691+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:56.863 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:56.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:56.863 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:56.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:56.863 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:56.487+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:56.861+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:56.861+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:57.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:56.861+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:57.469+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:57.469+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:57.469+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:57.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:58.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:57.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:58.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:57.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:57.727+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:57.727+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:57.727+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:58.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:26:58.699+0000 7f30555f0700 0 monclient(hunting): authenticate timed out after 300 2024-02-16T22:26:58.701 INFO:teuthology.orchestra.run.smithi002.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-16T22:26:58.848 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:58.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:58.851 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:58.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:58.851 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:58.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:58.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:58.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:58.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:58.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:58.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:58.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:59.186 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:58.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:59.186 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:58.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:59.187 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:58.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:26:59.615 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch ps' 2024-02-16T22:26:59.824 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:59.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:59.825 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:59.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:59.825 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:26:59.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:26:59.825 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:59.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:59.825 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:59.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:59.825 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:26:59.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:26:59.883 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/mon.smithi002/config 2024-02-16T22:27:00.079 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:26:59.953+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1411 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:00.079 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:59.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:00.080 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:59.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:00.080 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:26:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:26:59.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:00.805+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:00.805+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:00.805+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:00.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:00.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:00.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:00.734+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:00.734+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:00.734+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:01.642 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:01.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:01.642 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:01.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:01.642 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:01.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:01.826+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:01.826+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:01.826+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:01.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:01.701+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:01.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:01.701+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:01.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:01.701+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:02.851 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:02.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:02.852 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:02.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:02.852 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:02.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:02.852 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:02.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:02.852 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:02.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:02.852 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:02.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:02.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:02.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:02.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:03.879 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:03.651+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:03.880 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:03.651+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:03.880 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:03.651+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:03.880 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:03.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:03.880 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:03.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:03.880 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:03.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:03.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:04.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:03.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:04.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:03.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:04.924 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:04.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:04.924 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:04.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:04.924 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:04.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:04.925 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:04.655+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:04.925 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:04.655+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:04.925 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:04.655+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:05.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:04.954+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1491 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:05.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:04.922+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:05.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:04.922+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:05.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:04.922+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:05.624 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:05.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:05.624 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:05.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:05.625 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:05.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:05.896 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:05.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:05.896 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:05.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:05.897 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:05.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:05.895+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:05.895+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:05.895+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:06.943 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:06.614+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:06.944 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:06.614+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:06.944 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:06.614+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:06.944 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:06.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:06.944 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:06.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:06.944 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:06.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:07.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:06.942+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:07.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:06.942+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:07.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:06.942+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:07.959 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:07.661+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:07.959 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:07.661+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:07.959 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:07.661+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:07.959 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:07.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:07.959 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:07.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:07.959 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:07.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:07.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:07.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:07.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:08.434+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:08.434+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:08.434+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:08.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:08.684+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:08.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:08.684+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:08.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:08.684+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:08.929+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:09.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:08.929+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:09.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:08.929+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:09.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:09.432+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:09.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:09.432+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:09.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:09.432+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:09.667+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:09.667+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:09.667+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:09.955+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1567 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:09.934+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:09.934+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:09.934+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:10.617+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:10.617+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:10.617+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:10.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:10.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:10.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:10.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:10.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:10.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:11.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:11.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:11.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:11.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:11.652+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:11.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:11.652+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:11.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:11.652+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:11.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:12.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:11.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:12.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:11.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:12.649 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:12.509+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:12.650 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:12.509+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:12.650 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:12.509+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:12.940 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:12.648+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:12.941 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:12.648+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:12.941 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:12.648+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:12.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:12.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:13.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:12.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:13.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:13.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:13.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:13.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:13.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:13.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:13.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:13.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:13.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:14.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:13.959+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:14.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:13.959+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:14.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:13.959+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:14.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:14.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:14.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:14.428+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:14.428+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:14.428+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:15.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:14.956+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1645 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:14.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:14.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:14.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:15.625 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:15.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:15.626 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:15.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:15.626 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:15.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:15.877 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:15.624+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:15.877 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:15.624+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:15.877 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:15.624+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:16.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:15.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:16.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:15.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:16.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:15.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:16.672 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:16.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:16.672 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:16.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:16.672 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:16.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:16.671+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:16.671+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:16.671+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:16.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:16.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:16.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:16.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:16.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:16.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:17.656 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:17.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:17.656 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:17.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:17.657 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:17.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:17.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:17.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:17.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:17.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:17.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:17.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:17.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:17.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:17.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:18.684 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:18.415+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:18.684 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:18.415+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:18.684 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:18.415+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:18.683+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:18.683+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:18.683+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:18.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:18.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:18.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:18.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:18.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:18.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:19.713 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:19.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:19.714 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:19.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:19.714 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:19.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:19.969 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:19.956+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1721 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:19.970 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:19.773+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:19.970 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:19.773+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:19.970 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:19.773+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:19.970 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:19.711+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:19.970 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:19.711+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:19.970 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:19.711+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:20.677 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:20.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:20.677 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:20.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:20.678 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:20.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:20.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:20.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:20.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:20.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:20.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:20.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:20.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:20.676+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:20.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:20.676+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:20.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:20.676+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:21.679 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:21.400+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:21.680 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:21.400+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:21.680 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:21.400+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:21.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:21.778+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:21.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:21.778+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:21.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:21.778+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:21.678+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:21.678+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:21.678+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:22.723 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:22.372+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:22.723 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:22.372+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:22.723 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:22.372+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:22.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:22.800+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:22.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:22.800+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:22.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:22.800+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:22.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:22.721+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:22.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:22.721+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:22.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:22.721+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:23.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:23.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:23.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:23.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:24.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:23.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:24.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:23.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:23.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:23.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:23.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:24.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:24.355+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:24.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:24.355+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:24.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:24.355+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:24.985 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:24.957+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1801 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:24.985 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:24.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:24.985 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:24.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:24.985 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:24.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:24.986 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:24.818+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:24.986 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:24.818+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:24.986 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:24.818+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:25.699 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:25.396+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:25.699 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:25.396+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:25.699 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:25.396+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:25.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:25.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:25.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:25.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:25.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:25.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:25.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:25.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:25.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:25.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:25.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:25.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:26.408+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:26.408+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:26.408+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:27.222 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:26.866+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:27.222 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:26.866+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:27.222 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:26.866+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:27.223 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:26.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:27.223 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:26.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:27.223 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:26.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:27.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:27.385+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:27.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:27.385+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:27.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:27.385+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:27.714+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:27.714+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:27.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:27.714+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:27.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:27.834+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:27.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:27.834+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:27.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:27.834+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:28.376+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:28.376+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:28.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:28.376+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:28.743+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:29.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:28.743+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:29.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:28.743+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:29.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:28.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:29.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:28.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:29.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:28.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:29.698 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:29.328+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:29.698 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:29.328+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:29.698 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:29.328+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:29.961 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:29.916+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:29.961 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:29.916+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:29.961 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:29.916+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:29.961 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:29.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:29.961 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:29.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:29.961 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:29.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:30.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:29.957+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1877 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:30.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:30.293+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:30.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:30.293+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:30.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:30.293+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:30.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:30.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:30.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:30.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:30.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:30.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:30.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:30.878+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:30.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:30.878+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:30.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:30.878+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:31.643 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:31.303+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:31.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:31.303+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:31.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:31.303+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:31.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:31.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:31.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:31.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:31.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:31.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:32.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:32.294+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:32.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:32.294+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:32.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:32.294+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:32.835+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:32.835+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:32.835+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:32.736+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:32.736+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:32.736+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:33.721 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:33.315+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:33.721 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:33.315+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:33.721 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:33.315+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:33.855+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:33.855+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:33.855+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:33.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:33.719+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:33.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:33.719+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:33.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:33.719+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:34.291+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:34.291+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:34.291+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:35.086 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:34.959+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 1955 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:35.087 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:34.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:35.087 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:34.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:35.087 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:34.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:35.087 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:34.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:35.087 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:34.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:35.087 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:34.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:35.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:35.268+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:35.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:35.268+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:35.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:35.268+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:35.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:35.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:35.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:36.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:35.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:36.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:35.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:36.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:35.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:36.708 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:36.294+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:36.708 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:36.294+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:36.709 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:36.294+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:36.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:36.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:36.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:36.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:36.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:36.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:36.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:36.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:36.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:37.328+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:37.328+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:37.328+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:37.907+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:37.907+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:37.907+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:38.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:37.784+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:38.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:37.784+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:38.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:37.784+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:38.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:38.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:38.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:38.752+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:38.752+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:39.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:38.752+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:39.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:38.946+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:39.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:38.946+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:39.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:38.946+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:39.336+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:39.336+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:39.336+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:39.987 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:39.960+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2031 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:39.987 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:39.959+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:39.987 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:39.959+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:39.987 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:39.959+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:39.988 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:39.726+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:39.988 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:39.726+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:39.988 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:39.726+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:40.711 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:40.287+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:40.712 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:40.287+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:40.712 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:40.287+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:40.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:40.710+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:40.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:40.710+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:40.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:40.710+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:41.301 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:41.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:41.301 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:41.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:41.301 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:41.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:41.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:41.299+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:41.645 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:41.299+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:41.645 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:41.299+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:41.747+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:41.747+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:41.747+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:42.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:42.268+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:42.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:42.268+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:42.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:42.268+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:42.053+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:42.053+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:42.053+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:43.068 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:42.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:43.068 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:42.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:43.068 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:42.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:43.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:43.280+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:43.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:43.280+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:43.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:43.280+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:43.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:43.066+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:43.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:43.066+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:43.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:43.066+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:43.721+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:43.721+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:43.721+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:44.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:44.303+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:44.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:44.303+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:44.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:44.303+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:44.098+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:44.098+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:44.098+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:45.051 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:44.961+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2111 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:45.051 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:44.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:45.052 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:44.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:45.052 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:44.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:45.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:45.049+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:45.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:45.049+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:45.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:45.049+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:45.337+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:45.337+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:45.337+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:46.027 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:45.742+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:46.027 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:45.742+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:46.027 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:45.742+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:46.382 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:46.025+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:46.382 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:46.025+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:46.382 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:46.025+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:46.709 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:46.380+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:46.709 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:46.380+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:46.709 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:46.380+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:46.707+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:46.707+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:46.707+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:47.388 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:47.029+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:47.388 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:47.029+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:47.388 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:47.029+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:47.662 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:47.386+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:47.662 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:47.386+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:47.663 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:47.386+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:47.983 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:47.660+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:47.984 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:47.660+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:47.984 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:47.660+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:48.350 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:48.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:48.350 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:48.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:48.350 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:48.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:48.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:48.348+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:48.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:48.348+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:48.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:48.348+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:48.642+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:48.642+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:48.642+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:49.368 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:49.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:49.368 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:49.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:49.369 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:49.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:49.686 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:49.366+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:49.686 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:49.366+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:49.687 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:49.366+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:49.964 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:49.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:49.964 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:49.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:49.964 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:49.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:50.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:49.962+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2187 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:50.136+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:50.136+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:50.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:50.136+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:50.660 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:50.386+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:50.661 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:50.386+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:50.661 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:50.386+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:50.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:50.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:50.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:51.437 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:51.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:51.437 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:51.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:51.437 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:51.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:51.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:51.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:51.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:51.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:51.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:51.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:51.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:51.435+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:51.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:51.435+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:51.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:51.435+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:52.470 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:52.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:52.470 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:52.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:52.470 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:52.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:52.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:52.637+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:52.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:52.637+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:52.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:52.637+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:52.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:52.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:52.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:52.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:52.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:52.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:53.063+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:53.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:53.063+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:53.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:53.063+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:53.615+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:53.615+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:53.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:53.615+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:53.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:53.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:53.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:54.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:54.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:54.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:54.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:54.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:54.965 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:54.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:54.965 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:54.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:54.965 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:54.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:54.965 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:54.577+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:54.966 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:54.577+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:54.966 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:54.577+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:54.963+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2265 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:27:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:55.039+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:55.039+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:55.039+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:55.507+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:55.507+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:55.507+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:55.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:55.607+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:55.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:55.607+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:55.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:55.607+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:56.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:56.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:56.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:56.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:56.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:56.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:56.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:56.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:56.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:57.452 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:57.042+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:57.452 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:57.042+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:57.452 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:57.042+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:57.507+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:57.507+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:57.507+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:57.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:57.625+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:57.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:57.625+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:57.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:57.625+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:58.347 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:58.038+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:58.348 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:58.038+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:58.348 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:58.038+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:58.643+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:58.643+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:58.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:58.643+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:58.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:58.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:58.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:59.019+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:59.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:59.019+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:59.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:27:59.019+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:27:59.967 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:59.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:59.967 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:59.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:59.968 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:27:59.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:27:59.968 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:59.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:59.968 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:59.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:27:59.968 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:27:59.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:00.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:27:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:27:59.964+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2341 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:00.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:00.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:00.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:00.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:00.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:00.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:00.609+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:00.609+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:00.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:00.609+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:00.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:00.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:00.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:01.419 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:01.051+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:01.420 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:01.051+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:01.420 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:01.051+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:01.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:01.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:01.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:01.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:01.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:01.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:01.469+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:01.469+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:01.469+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:02.447 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:02.086+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:02.447 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:02.086+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:02.447 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:02.086+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:02.710 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:02.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:02.710 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:02.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:02.711 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:02.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:02.709+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:02.709+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:02.709+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:03.467 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:03.065+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:03.467 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:03.065+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:03.468 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:03.065+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:03.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:03.465+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:03.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:03.465+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:03.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:03.465+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:04.042 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:03.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:04.042 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:03.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:04.042 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:03.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:04.458 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:04.040+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:04.458 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:04.040+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:04.458 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:04.040+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:04.718 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:04.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:04.718 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:04.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:04.718 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:04.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:04.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:04.965+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2421 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:04.716+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:04.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:04.716+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:04.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:04.716+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:05.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:04.998+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:04.998+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:04.998+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:05.707 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:05.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:05.707 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:05.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:05.707 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:05.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:05.703+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:05.703+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:05.703+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:06.418 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:05.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:06.418 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:05.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:06.419 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:05.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:06.695 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:06.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:06.696 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:06.417+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:06.696 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:06.417+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:06.694+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:06.694+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:06.694+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:07.458 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:07.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:07.458 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:07.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:07.459 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:07.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:07.457+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:07.457+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:07.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:07.457+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:07.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:07.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:07.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:07.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:07.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:07.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:08.445 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:07.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:08.445 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:07.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:08.446 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:07.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:08.697 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:08.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:08.697 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:08.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:08.697 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:08.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:08.972 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:08.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:08.972 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:08.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:08.973 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:08.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:08.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:09.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:08.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:09.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:08.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:09.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:09.454+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:09.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:09.454+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:09.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:09.454+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:09.679+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:09.679+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:09.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:09.679+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:09.966+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2497 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:09.937+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:09.937+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:09.937+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:10.713 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:10.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:10.713 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:10.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:10.713 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:10.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:10.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:10.711+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:10.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:10.711+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:10.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:10.711+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:11.416 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:10.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:11.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:10.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:11.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:10.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:11.415+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:11.415+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:11.415+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:11.953+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:11.953+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:12.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:11.953+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:12.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:11.727+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:12.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:11.727+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:12.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:11.727+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:12.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:12.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:12.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:12.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:12.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:12.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:12.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:12.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:12.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:12.909+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:12.909+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:13.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:12.909+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:13.707 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:13.432+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:13.707 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:13.432+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:13.707 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:13.432+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:13.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:13.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:13.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:13.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:13.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:13.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:13.734+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:13.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:13.734+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:13.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:13.734+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:14.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:14.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:14.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:14.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:14.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:14.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:14.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:14.706+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:14.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:14.706+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:14.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:14.706+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:15.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:14.966+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2575 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:14.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:14.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:14.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:15.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:15.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:15.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:16.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:15.832+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:16.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:15.832+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:16.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:15.832+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:16.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:15.752+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:16.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:15.752+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:16.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:15.752+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:16.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:16.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:16.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:16.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:16.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:16.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:17.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:16.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:17.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:16.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:17.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:16.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:17.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:16.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:17.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:16.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:17.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:17.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:17.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:17.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:18.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:17.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:18.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:17.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:18.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:17.810+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:18.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:17.810+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:18.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:17.810+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:18.840 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:18.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:18.841 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:18.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:18.841 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:18.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:19.227 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:18.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:19.227 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:18.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:19.227 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:18.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:19.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:18.855+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:19.228 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:18.855+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:19.228 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:18.855+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:19.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:19.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:19.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:20.095 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:19.967+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2651 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:20.095 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:19.810+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:20.095 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:19.810+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:20.096 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:19.810+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:20.096 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:19.856+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:20.096 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:19.856+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:20.096 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:19.856+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:20.772 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:20.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:20.772 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:20.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:20.772 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:20.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:20.830+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:21.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:20.830+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:21.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:20.830+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:21.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:20.770+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:21.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:20.770+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:21.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:20.770+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:21.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:21.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:21.460+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:21.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:22.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:21.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:22.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:21.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:22.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:21.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:22.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:21.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:22.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:21.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:22.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:22.417+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:22.417+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:22.417+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:23.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:22.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:22.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:22.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:22.809+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:22.809+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:22.809+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:23.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:23.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:23.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:24.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:23.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:23.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:23.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:24.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:23.833+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:24.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:23.833+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:24.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:23.833+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:24.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:24.400+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:24.400+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:24.400+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:25.098 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:24.969+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2731 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:25.099 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:24.845+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:25.099 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:24.845+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:25.099 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:24.845+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:25.099 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:24.751+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:25.099 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:24.751+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:25.099 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:24.751+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:25.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:25.363+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:25.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:25.363+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:25.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:25.363+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:25.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:25.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:25.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:26.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:25.832+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:26.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:25.832+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:26.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:25.832+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:26.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:26.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:26.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:26.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:26.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:26.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:27.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:26.825+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:27.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:26.825+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:27.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:26.825+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:27.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:27.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:27.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:27.843+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:28.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:27.843+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:28.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:27.843+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:28.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:27.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:28.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:27.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:28.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:27.825+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:28.309+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:28.309+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:28.309+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:28.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:28.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:29.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:28.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:29.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:28.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:29.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:28.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:29.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:28.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:29.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:29.279+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:29.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:29.279+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:29.279+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:30.097 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:29.970+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2807 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:30.098 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:29.833+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:30.098 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:29.833+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:30.098 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:29.833+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:30.098 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:29.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:30.098 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:29.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:30.098 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:29.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:30.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:30.277+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:30.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:30.277+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:30.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:30.277+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:30.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:30.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:30.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:30.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:30.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:30.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:31.648 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:31.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:31.648 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:31.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:31.648 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:31.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:31.866+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:31.866+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:31.866+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:31.693+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:31.693+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:31.693+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:32.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:32.293+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:32.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:32.293+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:32.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:32.293+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:32.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:32.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:33.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:32.875+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:32.741+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:32.741+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:32.741+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:33.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:33.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:33.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:33.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:33.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:33.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:33.763+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:33.763+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:33.763+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:34.359+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:34.359+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:34.359+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:35.098 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:34.971+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2885 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:35.098 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:34.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:35.099 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:34.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:35.099 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:34.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:35.099 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:34.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:35.099 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:34.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:35.099 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:34.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:35.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:35.322+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:35.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:35.322+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:35.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:35.322+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:36.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:35.842+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:35.842+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:35.842+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:36.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:35.777+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:36.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:35.777+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:36.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:35.777+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:36.358+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:36.358+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:36.358+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:37.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:36.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:37.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:36.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:37.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:36.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:37.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:36.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:37.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:36.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:37.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:36.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:37.320+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:37.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:37.320+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:37.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:37.320+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:37.866+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:37.866+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:38.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:37.866+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:38.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:37.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:38.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:37.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:38.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:37.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:38.291+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:38.292+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:38.292+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:39.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:38.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:39.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:38.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:39.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:38.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:39.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:38.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:39.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:38.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:39.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:38.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:39.321+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:39.321+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:39.321+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:39.999 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:39.972+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 2961 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:39.999 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:39.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:39.999 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:39.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:39.999 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:39.746+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:39.999 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:39.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:40.000 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:39.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:40.000 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:39.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:40.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:40.371+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:40.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:40.371+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:40.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:40.371+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:40.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:40.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:41.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:40.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:41.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:40.798+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:41.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:40.798+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:41.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:40.798+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:41.648 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:41.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:41.648 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:41.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:41.649 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:41.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:41.756+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:41.756+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:41.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:41.756+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:41.693+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:41.693+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:41.693+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:42.691 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:42.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:42.691 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:42.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:42.691 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:42.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:42.718+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:42.718+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:42.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:42.718+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:42.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:42.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:42.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:42.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:42.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:42.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:43.698 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:43.393+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:43.698 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:43.393+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:43.698 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:43.393+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:43.703+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:44.068 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:43.703+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:44.069 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:43.703+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:44.069 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:43.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:44.069 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:43.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:44.069 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:43.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:44.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:44.422+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:44.422+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:44.422+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:44.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:44.667+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:44.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:44.667+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:44.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:44.667+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:44.999 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:44.973+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3041 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:45.000 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:44.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:45.000 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:44.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:45.000 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:44.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:45.628 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:45.397+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:45.628 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:45.397+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:45.628 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:45.397+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:45.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:45.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:45.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:45.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:45.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:45.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:45.626+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:45.626+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:45.626+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:46.654 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:46.349+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:46.654 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:46.349+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:46.654 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:46.349+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:46.763+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:46.763+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:46.763+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:46.652+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:46.652+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:46.652+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:47.634 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:47.358+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:47.635 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:47.358+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:47.635 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:47.358+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:47.805+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:47.805+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:47.805+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:47.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:47.633+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:47.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:47.633+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:47.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:47.633+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:48.673 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:48.389+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:48.673 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:48.389+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:48.673 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:48.389+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:48.762+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:48.762+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:48.762+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:48.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:48.671+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:48.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:48.671+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:48.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:48.671+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:49.627 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:49.357+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:49.627 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:49.357+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:49.627 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:49.357+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:49.625+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:49.625+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:49.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:49.625+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:49.749+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:49.749+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:49.749+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:50.362 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:49.974+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3117 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:50.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:50.587+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:50.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:50.587+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:50.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:50.587+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:50.361+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:50.361+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:50.361+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:50.780+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:50.780+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:50.780+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:51.616 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:51.350+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:51.616 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:51.350+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:51.616 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:51.350+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:51.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:51.614+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:51.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:51.614+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:51.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:51.614+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:51.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:51.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:51.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:51.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:51.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:51.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:52.606+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:52.606+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:52.606+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:52.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:52.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:52.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:52.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:52.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:52.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:52.774+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:52.774+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:52.774+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:53.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:53.577+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:53.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:53.577+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:53.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:53.577+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:53.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:53.425+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:53.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:53.425+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:53.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:53.425+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:53.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:54.239 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:53.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:54.239 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:53.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:54.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:54.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:54.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:54.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:54.380+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:54.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:54.380+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:54.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:54.380+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:55.102 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:54.975+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3195 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:28:55.102 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:54.809+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:55.103 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:54.809+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:55.103 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:54.809+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:55.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:55.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:55.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:55.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:55.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:55.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:55.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:55.383+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:55.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:55.383+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:55.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:55.383+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:55.784+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:55.784+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:55.784+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:56.715 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:56.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:56.716 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:56.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:56.716 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:56.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:56.716 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:56.420+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:56.716 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:56.420+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:56.716 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:56.420+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:56.782+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:56.782+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:56.782+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:57.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:57.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:57.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:57.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:57.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:57.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:57.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:57.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:57.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:57.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:57.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:58.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:58.519+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:58.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:58.519+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:58.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:58.519+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:58.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:58.470+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:58.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:58.470+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:58.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:58.470+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:58.824+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:58.824+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:58.824+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:28:59.787 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:59.473+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:59.787 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:59.473+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:59.788 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:28:59.473+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:28:59.788 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:59.475+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:59.788 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:59.475+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:28:59.788 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:28:59.475+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:00.105 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:28:59.975+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3271 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:00.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:59.792+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:00.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:59.792+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:00.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:28:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:28:59.792+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:00.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:00.463+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:00.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:00.463+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:00.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:00.463+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:00.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:00.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:00.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:00.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:01.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:00.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:01.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:00.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:01.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:01.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:01.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:01.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:01.488+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:01.727 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:01.488+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:01.727 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:01.488+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:01.792+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:01.792+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:01.792+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:02.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:02.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:02.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:02.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:02.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:02.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:02.406+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:02.406+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:02.406+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:02.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:02.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:02.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:03.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:03.439+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:03.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:03.439+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:03.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:03.439+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:03.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:03.486+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:03.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:03.486+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:03.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:03.486+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:04.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:03.826+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:04.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:03.826+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:04.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:03.826+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:04.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:04.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:04.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:04.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:04.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:04.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:04.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:04.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:04.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:04.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:04.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:04.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:05.105 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:04.976+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3350 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:05.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:04.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:05.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:04.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:05.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:04.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:05.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:05.470+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:05.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:05.470+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:05.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:05.470+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:05.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:05.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:05.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:06.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:05.838+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:06.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:05.838+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:05.838+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:06.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:06.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:06.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:06.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:06.401+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:06.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:06.401+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:06.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:06.401+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:06.824+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:06.824+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:06.824+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:07.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:07.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:07.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:07.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:07.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:07.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:07.409+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:07.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:07.409+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:07.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:07.409+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:08.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:07.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:08.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:07.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:08.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:07.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:08.562+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:08.562+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:08.562+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:08.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:08.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:08.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:08.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:08.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:08.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:09.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:08.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:09.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:08.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:09.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:08.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:09.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:09.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:09.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:09.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:09.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:09.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:09.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:09.392+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:09.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:09.392+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:09.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:09.392+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:10.104 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:09.978+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3430 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:10.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:09.836+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:10.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:09.836+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:10.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:09.836+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:10.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:10.364+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:10.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:10.364+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:10.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:10.364+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:10.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:10.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:10.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:10.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:11.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:10.821+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:11.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:10.822+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:11.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:10.822+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:11.651 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:11.322+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:11.651 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:11.322+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:11.651 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:11.322+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:11.652 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:11.517+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:11.652 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:11.517+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:11.652 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:11.517+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:11.776+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:11.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:11.776+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:11.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:11.776+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:12.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:12.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:12.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:12.817 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:12.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:12.817 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:12.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:12.817 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:12.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:13.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:12.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:13.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:12.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:13.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:12.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:13.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:13.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:13.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:13.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:13.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:13.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:14.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:13.851+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:14.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:13.851+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:14.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:13.851+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:14.340+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:14.340+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:14.340+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:14.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:14.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:14.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:15.147 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:14.979+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3508 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:15.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:14.870+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:15.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:14.870+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:15.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:14.870+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:15.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:15.349+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:15.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:15.349+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:15.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:15.349+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:15.827 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:15.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:15.827 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:15.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:15.827 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:15.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:15.825+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:15.825+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:15.825+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:16.372+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:16.372+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:16.372+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:16.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:16.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:16.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:17.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:16.837+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:17.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:16.837+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:17.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:16.837+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:17.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:17.372+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:17.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:17.372+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:17.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:17.372+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:17.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:17.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:17.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:17.827+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:17.827+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:17.827+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:18.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:18.325+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:18.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:18.325+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:18.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:18.325+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:18.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:18.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:18.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:18.778+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:18.778+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:18.778+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:19.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:19.375+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:19.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:19.375+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:19.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:19.375+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:19.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:19.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:19.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:19.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:19.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:19.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:20.107 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:19.980+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3590 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:20.107 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:19.823+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:20.107 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:19.823+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:20.107 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:19.823+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:20.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:20.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:20.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:20.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:20.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:20.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:20.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:20.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:20.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:20.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:20.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:20.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:20.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:20.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:20.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:21.651 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:21.369+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:21.651 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:21.369+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:21.651 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:21.369+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:21.652 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:21.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:21.652 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:21.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:21.652 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:21.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:21.804+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:21.804+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:21.804+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:22.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:22.412+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:22.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:22.412+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:22.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:22.412+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:22.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:22.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:22.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:23.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:22.782+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:23.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:22.782+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:23.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:22.782+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:23.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:23.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:23.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:23.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:23.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:23.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:23.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:23.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:23.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:24.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:23.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:24.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:23.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:24.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:23.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:24.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:24.484+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:24.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:24.484+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:24.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:24.484+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:24.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:24.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:24.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:25.107 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:24.981+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3677 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:25.107 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:24.839+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:25.107 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:24.839+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:25.107 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:24.839+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:25.830 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:25.524+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:25.830 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:25.524+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:25.830 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:25.524+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:25.830 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:25.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:25.830 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:25.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:25.831 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:25.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:25.828+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:25.828+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:26.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:25.828+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:26.834 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:26.494+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:26.834 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:26.494+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:26.835 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:26.494+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:26.835 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:26.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:26.835 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:26.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:26.835 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:26.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:26.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:26.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:26.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:27.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:27.446+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:27.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:27.446+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:27.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:27.446+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:27.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:27.501+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:27.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:27.501+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:27.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:27.501+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:27.856+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:27.856+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:27.856+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:28.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:28.450+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:28.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:28.450+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:28.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:28.450+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:28.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:28.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:28.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:28.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:28.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:28.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:28.865+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:28.865+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:28.865+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:29.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:29.418+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:29.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:29.418+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:29.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:29.418+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:29.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:29.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:29.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:30.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:29.982+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3754 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:29.879+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:29.879+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:29.879+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:30.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:30.406+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:30.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:30.406+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:30.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:30.406+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:30.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:30.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:30.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:30.890+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:30.890+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:30.890+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:31.652 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:31.391+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:31.652 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:31.391+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:31.652 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:31.391+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:31.652 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:31.534+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:31.652 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:31.534+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:31.652 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:31.534+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:31.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:31.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:31.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:32.387+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:32.387+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:32.387+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:32.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:32.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:32.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:32.887+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:32.887+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:32.887+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:33.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:33.390+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:33.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:33.390+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:33.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:33.390+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:33.499+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:33.499+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:33.499+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:34.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:33.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:34.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:33.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:34.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:33.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:34.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:34.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:34.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:34.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:34.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:34.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:34.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:34.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:35.109 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:34.983+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3843 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:35.109 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:34.853+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:35.109 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:34.853+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:35.110 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:34.853+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:35.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:35.406+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:35.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:35.406+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:35.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:35.406+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:35.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:35.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:35.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:35.888+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:35.888+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:35.888+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:36.448+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:36.448+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:36.448+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:36.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:36.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:36.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:36.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:36.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:36.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:36.841+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:36.841+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:36.841+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:37.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:37.408+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:37.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:37.408+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:37.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:37.408+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:37.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:37.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:37.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:37.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:37.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:37.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:38.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:37.872+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:38.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:37.872+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:38.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:37.872+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:38.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:38.363+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:38.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:38.363+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:38.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:38.363+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:38.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:38.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:38.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:38.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:38.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:38.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:38.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:38.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:38.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:39.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:39.380+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:39.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:39.380+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:39.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:39.380+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:39.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:39.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:39.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:40.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:39.984+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3920 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:40.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:39.904+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:39.904+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:39.904+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:40.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:40.348+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:40.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:40.348+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:40.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:40.348+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:40.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:40.504+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:40.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:40.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:40.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:40.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:40.860+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:40.860+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:40.860+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:41.653 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:41.369+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:41.653 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:41.369+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:41.653 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:41.369+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:41.653 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:41.529+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:41.653 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:41.529+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:41.653 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:41.529+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:41.833+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:41.833+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:41.833+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:42.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:42.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:42.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:42.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:42.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:42.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:42.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:42.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:42.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:43.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:42.841+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:43.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:42.841+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:43.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:42.841+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:43.529+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:43.529+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:43.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:43.529+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:43.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:43.312+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:43.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:43.312+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:43.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:43.312+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:43.874+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:43.874+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:43.874+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:44.527 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:44.267+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:44.527 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:44.267+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:44.527 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:44.267+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:44.907 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:44.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:44.907 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:44.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:44.907 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:44.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:45.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:44.986+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 3999 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:45.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:44.905+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:45.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:44.905+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:45.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:44.905+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:45.554 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:45.232+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:45.555 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:45.232+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:45.555 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:45.232+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:45.867 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:45.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:45.867 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:45.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:45.867 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:45.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:45.920+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:45.920+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:45.920+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:46.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:46.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:46.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:46.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:46.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:46.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:46.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:46.279+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:46.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:46.279+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:46.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:46.279+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:46.898+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:46.898+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:46.898+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:47.568 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:47.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:47.568 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:47.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:47.568 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:47.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:47.893 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:47.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:47.894 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:47.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:47.894 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:47.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:47.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:47.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:47.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:48.592 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:48.260+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:48.593 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:48.260+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:48.593 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:48.260+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:48.941 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:48.590+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:48.942 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:48.590+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:48.942 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:48.590+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:48.939+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:48.939+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:48.939+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:49.617 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:49.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:49.617 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:49.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:49.617 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:49.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:49.926 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:49.614+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:49.926 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:49.614+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:49.926 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:49.614+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:50.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:49.986+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4079 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:49.924+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:49.924+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:49.924+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:50.666 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:50.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:50.666 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:50.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:50.666 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:50.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:50.960 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:50.664+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:50.960 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:50.664+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:50.960 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:50.664+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:50.958+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:50.958+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:50.958+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:51.653 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:51.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:51.654 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:51.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:51.654 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:51.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:51.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:51.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:51.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:51.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:51.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:51.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:51.943+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:51.943+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:51.943+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:52.715 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:52.284+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:52.715 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:52.284+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:52.715 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:52.284+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:52.942+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:52.942+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:52.942+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:52.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:52.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:52.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:53.544 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:53.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:53.544 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:53.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:53.544 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:53.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:53.909 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:53.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:53.909 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:53.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:53.910 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:53.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:53.907+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:53.907+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:53.907+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:54.297+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:54.297+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:54.297+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:55.013 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:54.987+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4176 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:29:55.014 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:54.893+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:55.014 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:54.893+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:55.014 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:54.893+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:55.014 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:54.751+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:55.014 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:54.751+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:55.014 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:54.751+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:55.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:55.297+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:55.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:55.297+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:55.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:55.297+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:55.860+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:55.860+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:55.860+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:56.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:55.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:56.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:55.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:56.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:55.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:56.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:56.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:56.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:56.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:56.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:56.279+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:56.826+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:56.826+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:56.826+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:56.811+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:56.811+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:56.811+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:57.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:57.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:57.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:58.155 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:57.814+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:58.155 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:57.814+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:58.156 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:57.814+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:58.156 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:57.819+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:58.156 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:57.819+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:58.156 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:57.819+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:58.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:58.294+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:58.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:58.294+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:58.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:58.294+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:58.859+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:58.859+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:58.859+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:29:59.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:58.820+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:59.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:58.820+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:59.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:58.820+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:29:59.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:59.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:59.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:59.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:29:59.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:29:59.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:00.114 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:29:59.989+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4266 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:00.115 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:59.892+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:00.115 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:59.892+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:00.115 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:29:59.892+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:00.115 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:59.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:00.115 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:59.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:00.115 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:29:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:29:59.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:00.260+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:00.260+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:00.260+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:00.862+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:00.862+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:00.862+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:01.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:00.732+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:01.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:00.732+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:01.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:00.732+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:01.654 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:01.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:01.655 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:01.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:01.655 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:01.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:01.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:01.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:01.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:01.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:01.873+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:01.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:01.873+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:01.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:01.873+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:02.675 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:02.237+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:02.675 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:02.237+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:02.675 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:02.237+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:02.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:02.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:02.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:02.862+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:02.862+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:02.862+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:03.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:03.211+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:03.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:03.211+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:03.477 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:03.211+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:03.842 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:03.680+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:03.842 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:03.680+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:03.842 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:03.680+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:04.167 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:03.840+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:04.168 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:03.840+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:04.168 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:03.840+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:04.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:04.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:04.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:04.862+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:04.862+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:04.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:04.862+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:04.656+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:04.656+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:04.656+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:05.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:04.990+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4345 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:05.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:05.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:05.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:05.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:05.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:05.980 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:05.662+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:05.980 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:05.662+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:05.980 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:05.662+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:05.980 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:05.859+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:05.980 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:05.859+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:05.981 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:05.859+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:06.199+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:06.199+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:06.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:06.199+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:06.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:06.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:06.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:06.823+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:06.823+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:06.823+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:07.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:07.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:07.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:07.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:07.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:07.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:07.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:07.866+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:07.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:07.866+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:07.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:07.866+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:08.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:08.189+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:08.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:08.189+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:08.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:08.189+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:08.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:08.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:08.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:08.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:08.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:08.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:08.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:08.853+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:08.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:08.854+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:08.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:08.854+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:09.605 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:09.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:09.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:09.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:09.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:09.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:09.869 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:09.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:09.869 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:09.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:09.869 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:09.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:10.219 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:09.991+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4422 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:10.219 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:09.867+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:10.220 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:09.867+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:10.220 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:09.867+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:10.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:10.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:10.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:10.631+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:10.631+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:10.631+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:10.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:10.820+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:10.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:10.820+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:10.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:10.820+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:11.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:11.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:11.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:11.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:11.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:11.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:11.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:11.828+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:11.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:11.828+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:11.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:11.828+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:12.218+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:12.218+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:12.218+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:12.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:12.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:12.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:12.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:12.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:12.816+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:12.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:12.816+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:12.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:12.816+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:13.515 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:13.249+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:13.515 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:13.249+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:13.515 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:13.249+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:13.812 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:13.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:13.812 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:13.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:13.813 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:13.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:14.202 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:13.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:14.202 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:13.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:14.202 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:13.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:14.201+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:14.201+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:14.201+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:14.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:14.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:14.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:14.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:14.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:14.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:15.303 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:14.992+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4499 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:15.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:15.209+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:15.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:15.209+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:15.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:15.209+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:15.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:15.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:15.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:15.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:15.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:15.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:15.593+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:15.593+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:15.593+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:16.197+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:16.197+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:16.197+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:16.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:16.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:16.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:16.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:16.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:16.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:16.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:16.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:17.171+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:17.171+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:17.171+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:17.792 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:17.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:17.792 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:17.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:17.792 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:17.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:18.149 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:17.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:18.149 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:17.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:18.149 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:17.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:18.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:18.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:18.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:18.747+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:18.747+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:18.747+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:18.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:18.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:18.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:19.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:19.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:19.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:19.776 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:19.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:19.776 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:19.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:19.776 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:19.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:20.120 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:19.994+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4576 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:20.120 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:19.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:20.120 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:19.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:20.120 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:19.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:20.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:20.152+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:20.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:20.152+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:20.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:20.152+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:20.823 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:20.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:20.824 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:20.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:20.824 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:20.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:21.187 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:20.822+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:21.187 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:20.822+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:21.187 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:20.822+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:21.186+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:21.186+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:21.186+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:21.787 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:21.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:21.787 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:21.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:21.787 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:21.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:22.196 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:21.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:22.197 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:21.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:22.197 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:21.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:22.454 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:22.195+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:22.454 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:22.195+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:22.454 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:22.195+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:22.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:22.452+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:22.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:22.452+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:22.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:22.452+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:23.222 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:22.834+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:23.222 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:22.834+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:23.222 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:22.834+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:23.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:23.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:23.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:23.882 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:23.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:23.882 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:23.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:23.882 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:23.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:24.173 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:23.881+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:24.173 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:23.881+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:24.173 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:23.881+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:24.171+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:24.171+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:24.171+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:24.834 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:24.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:24.835 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:24.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:24.835 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:24.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:25.124 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:24.995+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4655 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:25.125 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:24.833+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:25.125 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:24.833+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:25.125 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:24.833+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:25.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:25.128+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:25.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:25.128+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:25.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:25.128+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:25.803 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:25.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:25.803 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:25.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:25.803 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:25.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:26.156 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:25.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:26.157 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:25.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:26.157 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:25.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:26.155+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:26.155+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:26.155+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:26.898 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:26.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:26.898 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:26.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:26.898 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:26.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:26.896+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:26.896+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:26.896+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:27.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:27.117+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:27.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:27.117+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:27.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:27.117+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:27.914 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:27.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:27.914 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:27.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:27.914 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:27.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:27.913+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:27.913+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:27.913+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:28.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:28.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:28.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:28.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:28.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:28.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:28.890 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:28.617+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:28.890 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:28.617+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:28.890 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:28.617+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:28.888+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:28.888+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:28.888+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:29.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:29.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:29.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:29.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:29.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:29.881+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:29.881+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:29.881+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:29.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:29.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:29.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:29.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:30.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:29.996+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4732 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:30.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:30.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:30.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:30.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:30.901 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:30.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:30.901 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:30.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:30.901 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:30.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:31.051+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:31.051+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:31.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:31.051+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:30.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:30.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:30.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:31.947 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:31.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:31.947 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:31.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:31.947 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:31.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:31.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:31.685+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:31.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:31.685+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:31.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:31.685+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:32.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:32.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:32.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:32.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:32.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:32.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:32.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:32.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:32.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:33.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:33.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:33.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:33.957 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:33.722+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:33.957 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:33.722+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:33.957 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:33.722+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:34.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:34.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:34.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:33.955+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:33.955+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:33.955+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:34.910+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:34.910+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:34.910+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:34.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:34.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:34.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:35.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:34.996+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4809 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:35.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:35.110+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:35.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:35.110+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:35.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:35.110+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:35.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:35.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:35.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:35.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:35.872+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:35.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:35.872+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:35.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:35.872+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:36.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:36.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:36.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:36.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:36.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:36.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:36.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:36.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:36.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:36.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:36.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:36.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:36.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:36.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:36.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:37.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:37.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:37.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:37.698+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:37.698+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:37.698+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:37.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:37.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:37.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:37.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:37.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:37.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:38.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:38.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:38.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:38.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:38.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:38.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:38.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:38.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:38.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:38.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:38.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:38.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:39.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:39.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:39.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:39.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:39.675+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:39.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:39.675+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:39.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:39.675+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:39.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:39.847+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:39.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:39.847+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:39.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:39.847+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:39.998+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4886 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:40.236+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:40.236+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:40.236+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:40.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:40.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:40.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:40.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:40.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:40.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:40.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:40.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:40.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:41.657 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:41.271+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:41.657 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:41.271+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:41.658 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:41.271+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:41.770+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:41.770+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:41.770+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:41.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:41.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:41.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:42.671 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:42.282+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:42.671 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:42.282+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:42.671 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:42.282+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:42.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:42.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:42.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:42.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:42.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:42.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:42.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:42.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:43.631 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:43.297+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:43.631 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:43.297+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:43.631 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:43.297+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:43.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:43.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:43.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:43.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:43.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:43.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:43.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:43.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:43.773+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:44.666 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:44.306+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:44.666 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:44.306+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:44.666 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:44.306+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:44.664+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:44.664+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:44.664+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:44.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:44.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:44.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:44.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:44.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:44.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:45.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:45.000+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 4965 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:45.656 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:45.354+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:45.656 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:45.354+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:45.656 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:45.354+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:45.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:45.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:45.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:45.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:45.686+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:45.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:45.686+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:45.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:45.686+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:46.674 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:46.399+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:46.674 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:46.399+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:46.674 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:46.399+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:46.674 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:46.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:46.674 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:46.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:46.674 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:46.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:46.672+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:46.672+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:46.672+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:47.702 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:47.383+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:47.702 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:47.383+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:47.702 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:47.383+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:47.702 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:47.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:47.702 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:47.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:47.702 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:47.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:47.700+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:47.700+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:47.700+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:48.655 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:48.361+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:48.655 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:48.362+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:48.655 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:48.362+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:48.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:48.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:48.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:48.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:48.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:48.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:49.686 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:49.343+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:49.687 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:49.343+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:49.687 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:49.343+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:49.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:49.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:49.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:49.685+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:49.685+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:49.685+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:50.340 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:50.001+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5042 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:50.718 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:50.339+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:50.718 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:50.339+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:50.718 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:50.339+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:50.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:50.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:50.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:50.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:50.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:50.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:51.658 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:51.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:51.658 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:51.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:51.658 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:51.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:51.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:51.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:51.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:51.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:51.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:51.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:51.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:51.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:52.696 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:52.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:52.696 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:52.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:52.696 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:52.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:52.694+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:52.694+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:52.694+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:52.728+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:52.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:52.728+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:52.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:52.728+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:53.685 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:53.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:53.685 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:53.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:53.685 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:53.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:53.707+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:53.707+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:53.707+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:53.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:53.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:53.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:54.716 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:54.344+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:54.716 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:54.344+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:54.717 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:54.344+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:54.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:54.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:54.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:54.714+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:54.714+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:54.714+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:55.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:30:55.002+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5119 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:30:55.668 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:55.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:55.668 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:55.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:55.669 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:55.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:55.667+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:55.667+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:55.667+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:55.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:55.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:55.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:55.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:55.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:55.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:56.623 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:56.347+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:56.623 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:56.347+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:56.623 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:56.347+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:56.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:56.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:56.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:56.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:56.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:56.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:57.383+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:57.383+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:57.383+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:57.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:57.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:57.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:57.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:57.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:57.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:57.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:58.693 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:58.428+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:58.694 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:58.428+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:58.694 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:58.428+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:58.694 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:58.591+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:58.694 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:58.591+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:58.694 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:58.591+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:58.692+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:58.692+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:58.692+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:30:59.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:59.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:59.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:59.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:59.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:30:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:30:59.384+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:30:59.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:59.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:59.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:59.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:30:59.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:30:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:30:59.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:00.005 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:59.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:00.005 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:59.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:00.006 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:30:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:30:59.730+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:00.340 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:00.003+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5196 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:00.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:00.339+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:00.606 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:00.339+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:00.607 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:00.339+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:00.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:00.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:00.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:00.758+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:00.758+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:00.758+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:01.597 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:01.292+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:01.597 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:01.292+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:01.597 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:01.292+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:01.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:01.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:01.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:01.596+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:01.596+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:01.596+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:02.625 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:02.276+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:02.626 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:02.276+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:02.626 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:02.276+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:02.752+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:02.752+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:02.752+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:02.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:02.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:02.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:03.547 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:03.248+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:03.547 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:03.248+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:03.547 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:03.248+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:03.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:03.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:03.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:03.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:03.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:03.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:04.215+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:04.215+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:04.215+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:04.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:04.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:04.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:04.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:04.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:04.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:05.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:05.004+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5275 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:05.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:05.219+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:05.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:05.219+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:05.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:05.219+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:05.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:05.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:05.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:05.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:05.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:05.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:05.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:05.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:06.185+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:06.185+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:06.185+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:06.656+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:06.656+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:06.656+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:06.639+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:06.639+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:06.639+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:07.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:07.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:07.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:07.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:07.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:07.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:07.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:07.633+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:07.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:07.633+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:07.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:07.633+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:08.167+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:08.167+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:08.167+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:08.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:08.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:08.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:08.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:08.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:08.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:08.636+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:08.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:08.636+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:08.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:08.636+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:09.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:09.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:09.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:09.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:09.629+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:09.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:09.629+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:09.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:09.629+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:09.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:09.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:09.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:10.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:10.006+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5352 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:10.134+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:10.134+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:10.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:10.134+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:10.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:10.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:10.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:10.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:10.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:10.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:10.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:10.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:10.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:11.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:11.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:11.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:11.883 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:11.592+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:11.883 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:11.592+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:11.883 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:11.592+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:11.883 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:11.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:11.883 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:11.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:11.883 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:11.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:12.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:12.498 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:12.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:12.498 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:12.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:12.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:12.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:12.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:12.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:12.558+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:12.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:12.558+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:12.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:12.558+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:13.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:13.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:13.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:13.559+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:13.559+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:13.559+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:13.508+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:13.508+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:13.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:13.508+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:14.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:14.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:14.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:14.955 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:14.489+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:14.955 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:14.489+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:14.955 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:14.489+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:14.955 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:14.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:14.955 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:14.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:14.955 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:14.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:15.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:15.007+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5429 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:15.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:15.133+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:15.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:15.133+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:15.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:15.133+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:15.535+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:15.535+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:15.535+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:15.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:15.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:15.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:16.091+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:16.091+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:16.091+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:16.560+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:16.560+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:16.560+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:16.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:16.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:16.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:17.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:17.098+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:17.098+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:17.098+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:17.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:17.521+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:17.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:17.521+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:17.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:17.521+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:17.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:17.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:17.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:17.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:18.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:18.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:18.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:18.557+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:18.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:18.557+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:18.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:18.557+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:18.498+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:18.498+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:18.498+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:19.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:19.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:19.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:19.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:19.591+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:19.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:19.591+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:19.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:19.591+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:19.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:19.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:19.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:20.007+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5506 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:20.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:20.086+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:20.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:20.086+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:20.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:20.086+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:20.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:20.517+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:20.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:20.517+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:20.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:20.517+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:20.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:20.572+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:20.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:20.572+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:20.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:20.572+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:21.068+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:21.068+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:21.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:21.068+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:21.595+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:21.595+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:21.595+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:21.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:21.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:21.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:22.050+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:22.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:22.050+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:22.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:22.050+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:22.576+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:22.576+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:22.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:22.576+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:22.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:22.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:22.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:22.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:22.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:22.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:23.034+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:23.034+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:23.034+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:23.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:23.610+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:23.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:23.610+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:23.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:23.610+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:23.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:23.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:23.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:24.059+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:24.059+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:24.059+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:24.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:24.628+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:24.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:24.628+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:24.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:24.628+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:24.639+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:24.639+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:24.639+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:25.303 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:25.008+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5585 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:25.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:25.062+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:25.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:25.062+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:25.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:25.062+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:25.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:25.607+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:25.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:25.607+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:25.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:25.607+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:25.680+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:25.680+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:25.680+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:26.041+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:26.041+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:26.041+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:26.637+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:26.637+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:26.637+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:26.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:26.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:26.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:27.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:27.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:27.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:27.668+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:27.668+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:27.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:27.668+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:27.740+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:27.740+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:27.740+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:28.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:28.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:28.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:28.676+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:28.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:28.676+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:28.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:28.676+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:28.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:28.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:28.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:29.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:29.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:29.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:29.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:29.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:29.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:29.700+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:29.700+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:29.700+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:30.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:30.011+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5662 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:30.020+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:30.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:30.020+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:30.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:30.020+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:30.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:30.686+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:30.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:30.687+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:30.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:30.687+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:30.740+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:30.740+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:30.740+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:31.047+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:31.047+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:31.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:31.047+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:31.650+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:31.650+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:31.650+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:31.724+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:31.724+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:31.724+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:32.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:32.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:32.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:32.602+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:32.602+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:32.602+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:32.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:32.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:32.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:33.387 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:33.018+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:33.387 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:33.018+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:33.387 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:33.018+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:33.556+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:33.556+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:33.556+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:33.999+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:33.999+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:33.999+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:34.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:33.798+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:34.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:33.798+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:34.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:33.798+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:34.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:34.559+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:34.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:34.559+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:34.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:34.559+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:34.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:34.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:34.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:35.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:35.012+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5739 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:35.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:34.988+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:35.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:34.988+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:35.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:34.988+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:35.543+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:35.543+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:35.543+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:36.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:35.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:36.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:35.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:36.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:35.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:36.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:35.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:36.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:35.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:36.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:35.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:36.533+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:36.533+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:36.533+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:37.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:37.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:37.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:37.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:37.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:37.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:37.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:36.783+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:37.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:36.783+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:37.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:36.783+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:37.794 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:37.494+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:37.794 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:37.494+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:37.794 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:37.494+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:37.971+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:37.971+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:37.971+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:38.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:37.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:38.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:37.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:38.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:37.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:38.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:38.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:38.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:38.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:38.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:38.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:38.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:38.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:38.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:39.954 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:39.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:39.954 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:39.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:39.954 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:39.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:39.954 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:39.544+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:39.954 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:39.544+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:39.954 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:39.544+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:40.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:40.014+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5816 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:40.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:39.952+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:40.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:39.952+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:40.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:39.952+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:40.703+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:40.703+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:40.703+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:40.573+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:40.573+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:40.573+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:40.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:40.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:41.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:40.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:41.695+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:41.695+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:41.695+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:41.594+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:41.594+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:41.594+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:42.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:42.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:42.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:42.618+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:42.618+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:42.618+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:42.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:42.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:42.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:42.979+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:42.979+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:42.979+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:43.580+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:43.580+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:43.580+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:43.768+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:43.768+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:43.768+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:44.025+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:44.025+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:44.025+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:44.626+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:44.626+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:44.626+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:44.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:44.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:44.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:45.303 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:45.015+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5895 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:45.029+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:45.029+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:45.029+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:45.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:45.610+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:45.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:45.610+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:45.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:45.610+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:45.734+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:45.734+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:45.734+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:46.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:45.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:46.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:45.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:46.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:45.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:46.972 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:46.567+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:46.972 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:46.567+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:46.972 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:46.567+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:46.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:46.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:46.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:46.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:46.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:46.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:46.971+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:46.971+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:46.971+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:47.595+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:47.595+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:47.595+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:47.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:47.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:47.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:48.019+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:48.019+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:48.019+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:48.614+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:48.614+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:48.614+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:48.722+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:48.722+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:48.722+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:48.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:48.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:48.982+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:49.658+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:49.658+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:49.658+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:49.729+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:49.729+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:49.729+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:50.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:50.016+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 5972 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:50.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:50.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:50.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:50.652+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:50.652+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:50.652+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:50.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:50.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:50.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:51.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:51.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:51.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:51.647+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:51.647+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:51.647+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:51.763+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:51.763+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:51.763+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:52.053+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:52.053+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:52.053+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:52.667+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:52.667+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:52.667+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:52.739+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:52.739+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:52.739+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:53.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:53.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:53.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:53.712+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:53.712+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:53.712+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:53.735+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:53.735+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:53.735+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:54.051+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:54.051+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:54.051+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:54.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:54.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:54.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:54.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:54.711+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:54.711+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:54.711+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:55.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:31:55.017+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6049 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:31:55.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:55.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:55.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:55.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:55.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:55.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:56.069 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:55.739+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:56.069 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:55.739+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:56.069 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:55.739+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:56.069 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:55.728+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:56.069 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:55.728+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:56.069 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:55.728+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:56.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:56.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:56.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:56.712+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:56.712+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:56.712+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:56.698+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:56.698+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:56.698+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:57.082+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:57.082+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:57.082+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:57.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:57.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:57.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:57.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:57.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:57.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:57.964 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:57.675+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:57.964 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:57.675+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:57.964 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:57.675+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:58.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:58.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:58.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:58.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:58.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:58.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:58.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:58.641+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:58.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:58.641+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:58.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:58.641+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:59.042+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:59.042+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:31:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:31:59.042+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:31:59.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:59.708+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:30.773304+0000 front 2024-02-16T22:21:30.773319+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:59.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:59.708+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:30.773351+0000 front 2024-02-16T22:21:30.773245+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:59.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:31:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:31:59.708+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:30.773282+0000 front 2024-02-16T22:21:30.773294+0000 (oldest deadline 2024-02-16T22:21:54.873199+0000) 2024-02-16T22:31:59.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:59.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:59.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:59.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:31:59.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:31:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:31:59.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:00.019+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6125 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:00.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:00.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:00.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:00.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:00.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:00.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:00.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:00.653+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:00.653+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:00.653+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:00.993+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:00.993+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:00.993+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:01.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-02-16T22:32:01.502+0000 7fccbad87700 0 monclient(hunting): authenticate timed out after 300 2024-02-16T22:32:01.506 INFO:teuthology.orchestra.run.smithi002.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-16T22:32:01.824 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:01.660+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:01.824 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:01.660+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:01.824 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:01.660+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:02.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:01.972+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:02.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:01.972+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:02.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:01.972+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-16T22:32:02.355 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch ps'" 2024-02-16T22:32:02.565 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=70403733bcde4eaab6618ab5a10968e1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch ps'" 2024-02-16T22:32:02.568 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-16T22:32:02.580 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-16T22:32:02.581 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:32:02.581 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-16T22:32:02.605 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:32:02.605 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-16T22:32:02.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-16T22:32:02.956 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:02.657+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:02.956 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:02.657+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:02.956 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:02.657+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:02.954+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:03.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:02.954+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:03.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:02.954+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:03.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:03.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:03.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:03.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:03.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:03.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:04.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:03.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:04.702+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:04.702+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:04.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:04.702+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:05.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:05.020+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6205 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:04.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:04.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:04.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:05.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:05.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:05.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:06.037+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:06.037+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:06.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:06.037+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:06.699+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:06.699+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:06.699+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:07.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:07.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:07.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:08.018 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:07.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:08.019 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:07.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:08.019 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:07.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:08.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:08.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:08.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:08.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:29.941729+0000 front 2024-02-16T22:21:29.941938+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:08.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:29.941705+0000 front 2024-02-16T22:21:29.941878+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:09.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:08.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:29.941744+0000 front 2024-02-16T22:21:29.941834+0000 (oldest deadline 2024-02-16T22:21:55.241525+0000) 2024-02-16T22:32:09.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:08.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 since back 2024-02-16T22:21:31.326630+0000 front 2024-02-16T22:21:31.326538+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:09.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:08.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-02-16T22:21:31.326579+0000 front 2024-02-16T22:21:31.326547+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:09.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:08.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 since back 2024-02-16T22:21:31.326559+0000 front 2024-02-16T22:21:31.326591+0000 (oldest deadline 2024-02-16T22:21:57.226506+0000) 2024-02-16T22:32:10.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:10.020+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6281 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:15.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:15.022+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6359 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:20.024+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6436 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:21.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:21.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:21.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:22.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:22.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:22.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:23.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:23.542+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:23.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:23.542+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:23.542+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:24.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:24.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:24.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:24.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:24.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:24.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:25.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:25.026+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6515 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:25.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:25.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:25.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:26.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:26.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:26.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:27.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:27.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:27.601+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:28.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:28.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:28.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:29.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:29.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:29.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:29.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:29.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:29.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:30.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:30.027+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6591 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:29.935+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:29.935+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:29.935+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:30.964 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:30.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:30.964 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:30.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:30.964 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:30.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:30.962+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:30.962+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:31.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:30.962+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:31.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:31.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:31.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:31.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:31.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:31.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:32.951 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:32.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:32.951 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:32.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:32.952 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:32.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:32.949+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:33.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:32.949+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:33.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:32.949+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:33.869 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:33.512+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:33.869 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:33.512+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:33.869 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:33.512+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:33.992+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:34.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:33.992+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:34.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:33.992+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:33.867+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:34.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:33.867+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:34.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:33.867+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:34.827 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:34.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:34.827 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:34.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:34.827 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:34.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:35.157 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:35.028+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6669 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:35.158 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:34.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:35.158 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:34.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:35.158 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:34.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:35.158 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:34.901+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:35.158 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:34.901+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:35.159 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:34.901+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:35.904 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:35.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:35.905 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:35.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:35.905 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:35.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:35.903+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:35.903+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:35.903+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:35.921+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:35.921+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:35.921+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:36.873 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:36.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:36.873 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:36.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:36.873 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:36.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:36.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:37.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:36.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:37.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:36.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:37.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:36.904+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:37.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:36.904+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:37.227 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:36.904+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:37.908 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:37.581+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:37.909 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:37.581+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:37.909 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:37.581+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:37.943+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:38.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:37.943+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:38.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:37.943+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:38.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:37.906+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:38.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:37.906+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:38.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:37.906+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:38.893 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:38.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:38.894 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:38.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:38.894 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:38.574+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:38.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:39.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:38.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:39.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:38.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:39.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:38.949+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:39.227 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:38.949+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:39.227 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:38.949+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:39.911 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:39.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:39.911 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:39.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:39.911 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:39.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:40.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:40.030+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6746 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:40.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:39.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:40.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:39.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:40.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:39.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:39.910+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:39.910+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:40.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:39.910+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:40.942 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:40.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:40.942 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:40.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:40.943 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:40.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:40.941+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:40.941+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:40.941+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:41.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:41.041+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:41.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:41.041+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:41.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:41.041+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:41.957 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:41.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:41.958 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:41.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:41.958 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:41.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:42.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:41.956+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:42.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:41.956+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:42.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:41.956+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:42.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:42.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:42.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:42.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:42.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:42.013+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:42.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:42.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:42.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:42.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:42.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:42.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:43.042+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:43.042+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:43.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:43.042+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:42.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:42.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:42.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:43.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:43.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:43.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:43.982+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:43.982+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:43.982+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:44.059+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:44.059+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:44.059+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:44.514+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:44.514+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:44.514+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:45.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:45.031+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6825 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:45.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:45.006+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:45.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:45.006+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:45.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:45.006+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:45.057+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:45.057+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:45.304 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:45.057+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:45.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:45.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:45.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:45.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:45.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:46.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:45.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:46.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:46.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:46.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:46.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:46.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:46.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:46.977+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:46.977+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:46.977+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:47.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:47.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:47.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:47.968 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:47.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:47.968 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:47.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:47.969 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:47.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:47.967+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:47.967+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:48.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:47.967+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:47.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:47.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:47.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:48.556+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:48.556+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:48.556+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:48.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:48.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:48.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:49.027+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:49.027+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:49.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:49.027+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:49.528+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:50.013 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:49.528+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:50.013 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:49.528+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:50.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:50.033+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6901 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:50.023+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:50.023+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:50.023+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:49.979+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:49.979+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:49.979+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:50.537+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:50.537+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:50.537+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:51.003+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:51.003+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:51.003+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:51.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:50.997+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:51.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:50.997+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:51.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:50.997+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:51.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:51.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:51.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:51.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:51.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:51.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:51.960+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:51.960+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:51.960+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:52.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:52.033+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:52.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:52.033+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:52.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:52.033+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:52.923 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:52.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:52.924 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:52.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:52.924 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:52.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:52.922+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:52.922+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:52.922+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:53.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:53.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:53.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:53.922 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:53.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:53.922 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:53.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:53.922 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:53.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:53.920+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:54.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:53.920+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:54.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:53.920+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:54.090+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:54.090+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:54.090+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:54.943 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:54.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:54.943 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:54.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:54.943 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:54.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:55.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:32:55.033+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 6979 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:32:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:54.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:54.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:54.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:55.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:55.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:55.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:55.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:55.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:55.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:55.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:55.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:55.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:56.331 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:55.975+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:56.331 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:55.975+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:56.331 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:55.975+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:56.331 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:56.163+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:56.332 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:56.163+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:56.332 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:56.163+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:56.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:56.501+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:56.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:56.501+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:56.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:56.501+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:57.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:57.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:57.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:57.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:56.937+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:57.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:56.937+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:57.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:56.937+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:57.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:57.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:57.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:58.191 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:57.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:58.192 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:57.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:58.192 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:57.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:58.460 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:58.189+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:58.460 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:58.189+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:58.460 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:58.189+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:58.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:58.458+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:58.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:58.458+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:58.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:58.458+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:59.403 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:59.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:59.404 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:59.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:59.404 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:32:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:32:59.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:32:59.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:58.978+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:59.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:58.978+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:59.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:58.978+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:32:59.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:59.451+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:59.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:59.451+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:32:59.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:32:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:32:59.451+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:00.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:00.035+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7057 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:00.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:59.939+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:00.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:59.939+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:00.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:32:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:32:59.939+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:00.149+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:00.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:00.149+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:00.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:00.149+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:00.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:00.442+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:00.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:00.442+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:00.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:00.442+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:01.182 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:00.911+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:01.183 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:00.911+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:01.183 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:00.911+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:01.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:01.408+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:01.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:01.408+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:01.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:01.408+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:01.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:01.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:01.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:01.883+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:01.883+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:01.883+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:02.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:02.225+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:02.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:02.225+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:02.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:02.225+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:02.368+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:02.368+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:02.368+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:02.906+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:03.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:02.906+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:03.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:02.906+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:03.555 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:03.225+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:03.555 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:03.225+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:03.555 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:03.225+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:03.555 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:03.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:03.555 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:03.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:03.556 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:03.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:03.952+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:03.952+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:03.952+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:04.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:04.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:04.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:04.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:04.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:04.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:04.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:04.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:04.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:04.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:04.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:04.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:05.235 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:05.037+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7136 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:05.236 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:04.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:05.236 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:04.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:05.236 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:04.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:05.547 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:05.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:05.547 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:05.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:05.547 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:05.476+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:05.548 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:05.234+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:05.548 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:05.234+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:05.548 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:05.234+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:06.277 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:05.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:06.277 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:05.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:06.277 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:05.998+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:06.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:06.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:06.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:06.275+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:06.275+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:06.275+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:07.285 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:06.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:07.286 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:06.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:07.286 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:06.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:07.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:07.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:07.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:07.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:07.284+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:07.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:07.284+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:07.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:07.284+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:07.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:07.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:08.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:07.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:08.619 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:08.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:08.619 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:08.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:08.620 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:08.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:08.620 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:08.263+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:08.620 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:08.263+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:08.620 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:08.263+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:08.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:08.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:08.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:09.513 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:09.233+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:09.513 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:09.233+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:09.514 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:09.233+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:09.926 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:09.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:09.926 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:09.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:09.926 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:09.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:10.038+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7212 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:10.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:09.924+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:10.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:09.924+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:10.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:09.924+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:10.498 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:10.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:10.498 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:10.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:10.498 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:10.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:10.969 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:10.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:10.969 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:10.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:10.969 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:10.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:10.967+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:10.967+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:11.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:10.967+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:11.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:11.209+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:11.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:11.209+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:11.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:11.209+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:11.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:11.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:11.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:12.219 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:11.951+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:12.219 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:11.951+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:12.219 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:11.951+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:12.217+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:12.217+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:12.217+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:12.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:12.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:12.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:12.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:12.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:12.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:13.222 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:12.926+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:13.223 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:12.926+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:13.223 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:12.926+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:13.221+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:13.221+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:13.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:13.221+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:13.885 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:13.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:13.885 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:13.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:13.885 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:13.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:14.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:13.883+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:14.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:13.883+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:14.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:13.883+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:14.261+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:14.261+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:14.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:14.261+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:14.465+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:14.465+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:14.465+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:15.042 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:14.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:15.042 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:14.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:15.042 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:14.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:15.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:15.040+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7290 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:15.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:15.269+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:15.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:15.269+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:15.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:15.269+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:15.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:15.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:15.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:16.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:15.815+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:16.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:15.815+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:16.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:15.815+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:16.244+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:16.244+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:16.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:16.244+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:16.440+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:16.440+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:16.440+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:17.216 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:16.801+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:17.216 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:16.801+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:17.216 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:16.801+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:17.214+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:17.214+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:17.214+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:17.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:17.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:17.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:17.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:17.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:17.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:18.175 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:17.752+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:18.176 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:17.752+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:18.176 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:17.752+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:18.174+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:18.174+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:18.174+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:18.776 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:18.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:18.776 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:18.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:18.777 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:18.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:19.146 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:18.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:19.147 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:18.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:19.147 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:18.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:19.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:19.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:19.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:19.814 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:19.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:19.814 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:19.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:19.814 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:19.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:20.067 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:20.041+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7367 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:20.068 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:19.812+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:20.068 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:19.812+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:20.068 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:19.812+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:20.434 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:20.144+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:20.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:20.144+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:20.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:20.144+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:20.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:20.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:20.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:20.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:20.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:20.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:21.110 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:20.852+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:21.110 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:20.852+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:21.110 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:20.852+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:21.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:21.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:21.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:21.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:21.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:21.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:21.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:21.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:21.468+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:22.142 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:21.870+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:22.142 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:21.870+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:22.142 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:21.870+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:22.442 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:22.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:22.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:22.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:22.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:22.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:22.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:22.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:22.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:22.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:22.910+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:22.910+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:22.910+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:23.091+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:23.091+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:23.091+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:23.920 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:23.482+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:23.921 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:23.482+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:23.921 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:23.482+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:23.919+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:24.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:23.919+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:24.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:23.919+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:24.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:24.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:24.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:24.964 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:24.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:24.964 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:24.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:24.964 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:24.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:25.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:25.042+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7446 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:24.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:24.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:24.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:25.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:25.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:25.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:25.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:25.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:25.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:25.944 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:25.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:25.944 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:25.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:25.944 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:25.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:25.942+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:26.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:25.942+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:26.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:25.942+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:26.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:26.032+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:26.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:26.032+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:26.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:26.032+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:26.515+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:26.515+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:26.515+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:26.990+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:26.990+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:27.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:26.990+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:27.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:26.984+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:27.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:26.984+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:27.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:26.984+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:27.504+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:27.504+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:27.504+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:27.984+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:28.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:27.984+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:28.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:27.984+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:28.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:27.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:28.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:27.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:28.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:27.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:28.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:28.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:28.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:29.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:29.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:29.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:29.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:28.985+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:29.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:28.985+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:29.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:28.985+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:29.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:29.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:29.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:29.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:29.946 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:29.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:30.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:30.044+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7522 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:29.944+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:29.944+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:29.944+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:30.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:30.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:30.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:30.970 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:30.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:30.971 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:30.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:30.971 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:30.477+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:30.969+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:30.969+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:30.969+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:31.012+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:31.012+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:31.012+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:31.429+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:31.429+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:31.429+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:31.931+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:31.931+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:31.931+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:32.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:32.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:32.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:32.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:32.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:32.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:32.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:32.401+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:32.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:32.401+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:32.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:32.401+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:32.966+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:32.966+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:33.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:32.966+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:33.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:33.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:33.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:33.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:33.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:33.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:34.443 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:33.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:34.444 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:33.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:34.444 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:33.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:34.444 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:34.063+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:34.444 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:34.063+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:34.445 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:34.063+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:34.442+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:34.442+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:34.442+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:35.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:35.045+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7600 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:35.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:35.024+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:35.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:35.024+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:35.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:35.024+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:35.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:35.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:35.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:35.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:35.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:35.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:35.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:35.407+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:35.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:35.407+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:35.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:35.407+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:36.442 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:36.032+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:36.442 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:36.032+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:36.443 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:36.032+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:36.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:36.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:36.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:36.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:36.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:36.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:36.440+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:36.440+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:36.440+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:37.404 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:37.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:37.404 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:37.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:37.404 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:37.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:37.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:37.079+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:37.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:37.079+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:37.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:37.079+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:37.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:37.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:37.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:38.411 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:38.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:38.411 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:38.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:38.411 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:38.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:38.411 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:37.994+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:38.411 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:37.994+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:38.412 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:37.994+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:38.409+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:38.409+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:38.409+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:39.445 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:39.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:39.445 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:39.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:39.446 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:39.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:39.446 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:39.041+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:39.446 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:39.041+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:39.446 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:39.041+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:39.442+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:39.442+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:39.442+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:40.046+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7677 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:40.088+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:40.088+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:40.088+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:40.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:40.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:40.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:40.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:40.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:40.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:41.074 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:40.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:41.074 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:40.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:41.442 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:41.055+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:41.442 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:41.055+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:41.442 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:41.055+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:41.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:41.039+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:41.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:41.039+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:41.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:41.039+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:41.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:41.440+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:41.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:41.440+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:41.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:41.440+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:42.411 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:42.100+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:42.411 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:42.100+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:42.411 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:42.100+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:42.411 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:42.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:42.412 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:42.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:42.412 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:42.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:42.488+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:42.488+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:42.488+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:43.141+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:43.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:43.141+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:43.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:43.141+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:43.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:43.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:43.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:43.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:43.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:43.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:43.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:44.167+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:44.167+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:44.167+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:44.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:44.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:44.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:44.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:44.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:44.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:45.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:45.048+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7756 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:45.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:45.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:45.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:45.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:45.038+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:45.304 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:45.038+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:45.304 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:45.038+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:45.486+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:45.486+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:45.486+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:46.484 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:46.129+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:46.485 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:46.129+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:46.485 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:46.129+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:46.485 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:45.997+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:46.485 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:45.997+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:46.485 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:45.997+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:46.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:46.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:46.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:47.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:47.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:47.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:47.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:47.007+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:47.007+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:47.007+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:47.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:47.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:47.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:48.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:48.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:48.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:48.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:48.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:48.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:48.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:48.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:48.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:48.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:48.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:48.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:48.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:48.991+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:48.991+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:48.991+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:49.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:49.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:49.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:49.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:49.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:49.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:50.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:50.050+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7832 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:50.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:50.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:50.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:49.974+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:49.974+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:50.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:49.974+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:50.931 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:50.486+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:50.931 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:50.486+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:50.931 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:50.486+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:51.184 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:50.929+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:51.185 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:50.929+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:51.185 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:50.929+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:51.182+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:51.182+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:51.182+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:51.950 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:51.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:51.950 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:51.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:51.951 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:51.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:52.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:52.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:52.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:51.948+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:52.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:51.948+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:52.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:51.948+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:52.924 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:52.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:52.925 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:52.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:52.925 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:52.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:53.207 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:52.923+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:53.207 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:52.923+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:53.208 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:52.923+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:53.205+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:53.205+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:53.205+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:53.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:53.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:53.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:53.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:53.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:53.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:54.193+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:54.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:54.193+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:54.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:54.193+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:53.945+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:53.945+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:53.945+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:54.944 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:54.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:54.945 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:54.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:54.945 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:54.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:55.224 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:33:55.051+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7910 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:33:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:54.943+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:54.943+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:54.943+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:55.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:55.222+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:55.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:55.222+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:55.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:55.222+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:55.905 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:55.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:55.906 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:55.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:55.906 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:55.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:55.904+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:55.904+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:55.904+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:56.565 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:56.239+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:56.565 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:56.239+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:56.565 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:56.239+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:56.909 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:56.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:56.909 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:56.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:56.910 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:56.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:56.908+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:56.908+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:56.908+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:57.582 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:57.261+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:57.583 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:57.261+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:57.583 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:57.261+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:57.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:57.581+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:57.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:57.581+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:57.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:57.581+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:57.946+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:57.946+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:57.946+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:58.574 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:58.240+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:58.574 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:58.240+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:58.574 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:58.240+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:58.971 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:58.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:58.972 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:58.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:58.972 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:58.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:58.969+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:58.969+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:58.969+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:33:59.617 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:59.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:59.617 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:59.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:59.617 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:33:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:33:59.290+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:33:59.953 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:59.615+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:59.953 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:59.615+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:33:59.953 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:33:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:33:59.615+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:00.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:00.053+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 7987 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:59.952+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:59.952+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:33:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:33:59.952+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:00.628 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:00.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:00.628 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:00.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:00.628 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:00.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:00.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:00.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:00.949 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:00.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:00.949 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:00.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:00.946+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:00.946+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:00.946+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:01.665 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:01.232+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:01.665 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:01.232+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:01.665 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:01.232+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:01.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:01.664+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:01.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:01.664+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:01.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:01.664+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:01.971+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:01.971+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:01.971+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:02.621 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:02.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:02.621 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:02.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:02.621 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:02.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-16T22:34:02.745 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-02-16T22:34:02.745 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:34:02.745 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-16T22:34:02.765 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-02-16T22:34:02.765 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-16T22:34:02.824 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-16T22:34:02.891 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:02.619+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:02.891 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:02.619+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:02.891 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:02.619+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:02.957+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:02.957+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:02.957+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:03.558 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:03.254+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:03.559 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:03.254+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:03.559 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:03.254+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:03.915 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:03.588+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:03.915 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:03.588+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:03.915 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:03.588+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:04.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:03.924+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:04.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:03.924+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:04.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:03.924+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:04.592 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:04.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:04.592 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:04.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:04.592 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:04.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:04.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:04.590+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:04.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:04.590+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:04.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:04.590+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:05.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:05.054+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8063 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:05.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:04.937+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:05.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:04.937+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:05.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:04.937+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:05.638 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:05.301+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:05.638 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:05.301+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:05.638 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:05.301+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:05.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:05.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:05.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:05.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:05.929 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:05.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:06.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:05.926+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:06.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:05.926+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:05.926+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:06.651 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:06.268+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:06.651 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:06.268+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:06.652 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:06.268+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:06.972 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:06.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:06.972 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:06.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:06.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:06.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:06.971+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:06.971+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:06.971+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:07.646 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:07.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:07.646 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:07.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:07.646 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:07.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:07.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:07.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:07.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:07.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:07.941 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:07.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:08.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:07.939+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:08.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:07.939+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:08.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:07.939+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:08.661 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:08.303+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:08.661 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:08.303+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:08.661 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:08.303+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:08.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:08.659+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:08.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:08.659+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:08.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:08.659+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:09.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:08.938+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:09.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:08.938+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:09.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:08.938+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:09.673 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:09.351+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:09.674 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:09.351+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:09.674 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:09.351+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:09.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:09.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:09.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:09.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:09.937 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:09.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:10.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:10.056+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8130 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:10.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:09.936+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:10.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:09.936+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:10.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:09.936+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:10.668 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:10.382+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:10.668 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:10.382+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:10.668 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:10.382+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:10.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:10.909+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:10.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:10.909+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:10.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:10.909+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:10.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:10.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:10.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:11.672 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:11.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:11.696 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:11.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:11.696 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:11.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:11.697 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:11.408+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:11.697 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:11.408+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:11.697 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:11.408+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:11.887+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:11.887+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:11.887+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:12.699 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:12.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:12.699 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:12.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:12.699 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:12.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:12.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:12.906+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:12.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:12.906+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:12.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:12.906+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:12.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:12.697+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:12.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:12.697+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:12.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:12.697+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:13.708 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:13.450+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:13.709 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:13.450+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:13.709 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:13.450+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:13.895+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:13.895+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:13.895+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:13.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:13.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:13.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:14.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:14.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:14.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:15.059 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:14.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:15.059 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:14.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:15.059 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:14.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:15.060 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:14.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:15.060 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:14.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:15.060 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:14.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:15.432 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:15.057+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8199 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:15.690 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:15.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:15.691 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:15.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:15.691 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:15.431+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:15.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:15.929+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:15.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:15.929+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:15.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:15.929+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:15.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:15.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:15.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:16.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:16.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:16.449+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:17.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:16.976+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:17.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:16.976+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:17.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:16.976+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:17.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:16.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:17.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:16.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:17.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:16.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:17.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:17.442+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:17.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:17.442+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:17.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:17.442+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:17.958+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:17.958+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:17.958+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:18.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:17.766+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:18.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:17.766+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:18.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:17.766+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:18.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:18.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:18.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:18.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:18.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:18.471+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:19.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:19.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:19.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:19.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:18.810+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:19.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:18.810+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:19.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:18.810+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:19.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:19.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:19.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:19.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:19.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:19.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:20.061 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:20.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:20.061 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:20.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:20.061 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:20.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:20.061 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:19.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:20.061 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:19.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:20.062 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:19.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:20.059+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8267 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:20.490+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:20.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:20.490+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:20.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:20.490+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:20.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:20.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:20.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:21.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:21.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:21.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:21.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:21.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:21.529+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:21.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:21.529+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:21.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:21.529+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:21.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:21.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:21.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:22.346 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:22.039+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:22.346 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:22.039+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:22.347 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:22.039+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:22.670 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:22.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:22.670 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:22.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:22.670 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:22.527+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:22.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:22.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:22.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:23.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:23.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:23.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:23.531+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:23.531+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:23.531+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:23.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:23.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:23.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:24.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:24.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:24.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:24.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:24.565+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:24.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:24.565+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:24.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:24.565+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:24.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:24.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:24.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:25.303 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:25.059+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8340 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:25.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:25.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:25.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:25.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:25.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:25.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:25.674 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:25.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:25.674 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:25.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:25.675 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:25.532+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:25.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:25.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:25.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:25.999+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:25.999+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:25.999+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:26.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:26.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:26.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:26.714+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:26.714+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:26.714+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:27.002+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:27.002+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:27.002+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:27.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:27.520+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:27.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:27.520+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:27.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:27.520+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:27.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:27.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:27.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:28.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:28.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:28.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:28.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:28.543+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:28.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:28.543+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:28.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:28.543+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:28.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:28.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:28.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:29.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:29.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:29.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:29.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:29.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:29.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:29.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:29.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:29.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:29.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:29.696+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:29.696+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:29.696+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:30.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:30.061+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8413 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:30.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:30.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:30.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:30.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:30.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:30.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:30.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:30.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:30.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:30.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:30.709+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:30.709+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:30.709+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:31.132+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:31.132+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:31.132+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:31.535+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:31.535+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:31.535+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:31.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:31.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:31.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:32.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:32.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:32.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:32.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:32.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:32.565+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:32.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:32.565+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:32.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:32.565+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:32.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:32.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:32.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:32.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:33.103+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:33.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:33.103+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:33.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:33.103+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:33.553+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:33.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:33.553+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:33.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:33.553+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:33.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:33.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:33.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:33.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:33.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:34.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:34.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:34.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:34.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:34.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:34.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:34.550+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:34.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:34.550+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:34.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:34.550+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:34.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:34.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:34.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:34.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:34.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:35.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:35.064+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8491 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:35.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:35.147+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:35.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:35.147+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:35.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:35.147+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:35.589+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:35.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:35.589+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:35.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:35.589+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:35.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:35.636+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:35.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:35.636+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:35.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:35.636+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:36.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:36.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:36.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:36.614+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:36.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:36.614+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:36.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:36.614+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:36.613+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:36.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:36.613+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:36.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:36.613+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:37.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:37.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:37.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:37.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:37.632+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:37.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:37.633+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:37.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:37.633+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:37.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:37.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:37.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:38.158+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:38.158+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:38.158+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:38.640+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:38.640+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:38.640+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:38.586+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:38.586+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:38.586+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:39.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:39.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:39.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:39.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:39.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:39.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:39.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:39.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:39.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:39.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:39.622+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:39.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:39.622+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:39.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:39.622+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:40.065+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8567 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:40.153+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:40.153+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:40.153+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:40.780 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:40.632+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:40.780 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:40.632+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:40.780 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:40.632+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:40.780 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:40.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:40.780 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:40.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:40.781 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:40.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:41.473 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:41.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:41.473 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:41.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:41.473 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:41.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:41.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:41.629+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:41.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:41.629+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:41.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:41.629+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:41.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:41.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:41.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:41.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:41.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:41.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:42.447 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:42.130+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:42.447 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:42.130+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:42.447 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:42.130+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:42.673+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:42.673+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:42.673+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:42.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:42.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:42.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:42.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:42.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:42.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:43.421 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:43.166+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:43.421 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:43.166+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:43.421 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:43.166+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:43.685 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:43.419+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:43.685 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:43.419+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:43.685 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:43.419+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:43.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:43.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:43.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:43.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:44.460 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:44.135+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:44.461 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:44.135+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:44.461 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:44.135+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:44.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:44.694+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:44.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:44.694+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:44.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:44.694+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:44.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:44.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:44.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:44.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:45.420 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:45.067+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8646 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:45.420 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:45.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:45.420 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:45.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:45.421 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:45.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:45.682 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:45.418+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:45.682 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:45.418+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:45.682 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:45.418+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:45.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:45.680+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:45.680+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:45.680+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:46.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:46.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:46.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:46.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:46.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:46.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:46.723 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:46.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:46.723 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:46.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:46.723 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:46.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:46.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:46.722+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:46.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:46.722+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:46.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:46.722+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:47.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:47.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:47.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:47.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:47.673+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:47.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:47.673+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:47.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:47.673+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:47.498+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:47.498+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:47.498+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:48.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:48.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:48.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:48.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:48.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:48.687+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:48.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:48.687+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:48.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:48.687+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:48.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:48.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:48.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:49.166+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:49.166+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:49.166+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:49.707+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:49.707+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:49.707+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:49.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:49.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:49.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:50.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:50.068+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8722 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:50.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:50.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:50.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:50.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:50.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:50.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:50.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:50.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:50.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:50.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:50.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:50.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:51.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:51.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:51.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:51.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:51.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:51.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:51.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:51.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:51.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:51.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:51.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:51.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:51.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:52.204+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:52.204+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:52.204+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:52.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:52.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:52.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:52.535+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:52.733+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:52.733+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:52.733+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:53.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:53.251+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:53.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:53.251+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:53.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:53.251+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:53.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:53.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:53.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:53.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:53.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:53.687+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:53.687+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:53.687+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:54.514 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:54.228+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:54.515 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:54.228+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:54.515 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:54.228+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:54.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:54.512+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:54.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:54.512+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:54.513+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:54.645+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:54.645+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:54.645+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:55.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:34:55.070+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8799 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:34:55.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:55.245+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:55.245+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:55.245+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:55.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:55.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:55.551+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:55.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:55.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:55.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:55.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:55.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:55.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:56.267+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:56.267+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:56.267+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:56.613+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:56.613+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:56.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:56.613+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:56.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:56.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:56.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:57.604 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:57.301+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:57.605 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:57.301+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:57.605 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:57.301+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:57.623+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:57.623+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:57.623+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:57.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:57.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:57.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:58.648 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:58.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:58.648 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:58.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:58.648 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:58.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:58.649 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:58.307+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:58.649 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:58.307+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:58.649 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:58.307+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:58.647+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:58.647+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:58.647+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:59.684 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:59.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:59.684 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:59.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:59.685 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:34:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:34:59.565+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:34:59.685 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:59.349+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:59.685 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:59.349+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:59.685 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:34:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:34:59.349+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:34:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:59.682+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:59.682+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:34:59.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:34:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:34:59.682+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:00.072+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8877 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:00.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:00.311+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:00.311+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:00.311+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:00.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:00.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:00.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:00.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:00.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:00.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:01.609 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:01.284+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:01.609 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:01.284+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:01.609 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:01.284+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:01.699+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:01.699+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:01.699+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:01.607+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:01.607+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:01.607+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:02.581 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:02.271+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:02.581 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:02.271+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:02.582 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:02.271+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:02.705+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:02.705+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:02.705+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:02.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:02.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:02.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:03.562 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:03.285+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:03.562 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:03.285+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:03.563 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:03.285+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:03.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:03.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:03.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:03.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:03.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:03.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:03.656+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:03.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:03.656+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:03.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:03.656+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:04.559 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:04.297+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:04.559 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:04.297+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:04.559 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:04.297+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:04.694+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:04.694+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:04.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:04.694+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:04.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:04.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:04.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:05.355 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:05.073+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 8955 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:05.355 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:05.291+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:05.355 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:05.291+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:05.356 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:05.291+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:05.647 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:05.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:05.647 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:05.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:05.648 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:05.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:05.646+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:05.646+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:05.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:05.646+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:06.600 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:06.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:06.600 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:06.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:06.601 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:06.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:06.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:06.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:06.599+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:06.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:06.654+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:06.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:06.654+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:06.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:06.654+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:07.596 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:07.328+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:07.597 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:07.328+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:07.597 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:07.328+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:07.595+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:07.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:07.595+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:07.595+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:07.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:07.640+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:07.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:07.640+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:07.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:07.640+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:08.640 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:08.343+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:08.641 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:08.343+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:08.641 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:08.343+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:08.641 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:08.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:08.641 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:08.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:08.641 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:08.558+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:08.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:08.639+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:08.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:08.639+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:08.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:08.639+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:09.582 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:09.310+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:09.582 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:09.310+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:09.582 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:09.310+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:09.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:09.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:09.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:09.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:09.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:09.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:09.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:09.686+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:09.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:09.686+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:09.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:09.686+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:10.343 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:10.075+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9032 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:10.605 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:10.343+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:10.605 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:10.343+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:10.605 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:10.343+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:10.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:10.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:10.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:10.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:10.638+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:10.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:10.638+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:10.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:10.638+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:11.646 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:11.304+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:11.646 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:11.304+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:11.646 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:11.304+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:11.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:11.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:11.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:11.661+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:11.661+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:11.661+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:12.655 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:12.289+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:12.656 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:12.289+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:12.656 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:12.289+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:12.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:12.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:12.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:12.697+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:12.697+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:12.697+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:13.562 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:13.275+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:13.563 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:13.275+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:13.563 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:13.275+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:13.661+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:13.661+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:13.661+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:13.700+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:13.700+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:13.700+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:14.571 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:14.279+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:14.571 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:14.279+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:14.571 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:14.279+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:14.666+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:14.666+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:14.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:14.666+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:14.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:14.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:14.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:15.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:15.076+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9109 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:15.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:15.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:15.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:15.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:15.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:15.641+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:15.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:15.641+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:15.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:15.641+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:15.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:15.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:15.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:16.605 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:16.230+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:16.605 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:16.230+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:16.605 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:16.230+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:16.603+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:16.603+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:16.603+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:16.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:16.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:16.769+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:17.566 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:17.232+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:17.566 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:17.232+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:17.566 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:17.232+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:17.563+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:17.563+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:17.563+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:17.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:17.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:17.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:18.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:18.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:18.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:18.522+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:18.522+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:18.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:18.522+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:18.717+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:18.717+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:18.717+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:19.503 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:19.230+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:19.503 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:19.230+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:19.504 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:19.230+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:19.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:19.501+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:19.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:19.501+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:19.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:19.501+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:19.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:19.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:19.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:20.077+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9187 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:20.235+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:20.235+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:20.235+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:20.512+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:20.512+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:20.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:20.512+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:20.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:20.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:20.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:21.668 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:21.520+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:21.668 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:21.520+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:21.669 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:21.520+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:21.669 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:21.271+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:21.669 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:21.271+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:21.669 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:21.271+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:21.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:21.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:21.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:22.633 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:22.483+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:22.633 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:22.483+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:22.634 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:22.483+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:22.634 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:22.254+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:22.634 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:22.254+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:22.634 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:22.254+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:22.631+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:22.631+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:22.631+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:23.207+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:23.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:23.207+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:23.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:23.207+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:23.787 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:23.677+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:23.787 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:23.677+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:23.787 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:23.677+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:23.787 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:23.481+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:23.788 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:23.481+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:23.788 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:23.481+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:24.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:24.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:24.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:24.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:24.708+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:24.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:24.708+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:24.708+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:24.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:24.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:24.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:24.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:24.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:24.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:25.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:25.080+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9265 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:25.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:25.211+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:25.211+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:25.211+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:25.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:25.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:25.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:25.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:25.477+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:25.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:25.477+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:25.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:25.477+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:26.210+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:26.210+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:26.210+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:26.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:26.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:26.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:26.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:26.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:26.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:26.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:26.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:26.498+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:27.177+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:27.177+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:27.177+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:27.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:27.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:27.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:27.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:27.506+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:27.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:27.506+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:27.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:27.506+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:28.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:28.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:28.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:28.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:28.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:28.620+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:28.620+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:28.620+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:28.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:28.517+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:28.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:28.517+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:28.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:28.517+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:29.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:29.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:29.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:29.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:29.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:29.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:29.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:29.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:29.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:29.526+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:29.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:29.526+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:29.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:29.526+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:30.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:30.082+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9342 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:30.146+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:30.146+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:30.146+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:30.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:30.576+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:30.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:30.576+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:30.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:30.576+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:30.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:30.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:30.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:31.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:31.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:31.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:31.591+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:31.591+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:31.591+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:31.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:31.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:31.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:32.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:32.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:32.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:32.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:32.578+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:32.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:32.578+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:32.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:32.578+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:32.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:32.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:32.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:33.153+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:33.153+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:33.153+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:33.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:33.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:33.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:33.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:33.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:33.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:34.192+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:34.192+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:34.192+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:34.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:34.534+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:34.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:34.534+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:34.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:34.534+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:34.695+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:34.695+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:34.695+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:35.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:35.083+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9419 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:35.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:35.226+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:35.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:35.226+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:35.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:35.226+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:35.503+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:35.503+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:35.503+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:35.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:35.647+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:35.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:35.647+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:35.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:35.647+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:36.467 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:36.179+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:36.467 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:36.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:36.467 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:36.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:36.465+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:36.465+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:36.465+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:36.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:36.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:36.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:36.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:36.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:36.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:37.148+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:37.148+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:37.148+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:37.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:37.493+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:37.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:37.493+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:37.493+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:37.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:37.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:37.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:38.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:38.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:38.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:38.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:38.534+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:38.534+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:38.534+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:38.675+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:38.675+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:38.675+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:39.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:39.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:39.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:39.561+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:39.561+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:39.561+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:39.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:39.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:39.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:40.085+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9497 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:40.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:40.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:40.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:40.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:40.526+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:40.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:40.526+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:40.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:40.526+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:40.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:40.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:40.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:41.203+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:41.203+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:41.203+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:41.556+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:41.556+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:41.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:41.556+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:41.697+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:41.697+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:41.697+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:42.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:42.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:42.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:42.553+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:42.553+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:42.553+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:42.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:42.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:42.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:42.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:42.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:43.509 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:43.246+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:43.509 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:43.246+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:43.509 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:43.246+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:43.507+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:43.507+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:43.507+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:43.698+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:43.698+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:43.698+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:44.558 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:44.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:44.559 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:44.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:44.559 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:44.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:44.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:44.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:44.557+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:44.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:44.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:44.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:44.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:45.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:45.087+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9575 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:45.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:45.237+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:45.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:45.237+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:45.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:45.237+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:45.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:45.519+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:45.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:45.519+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:45.519+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:45.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:45.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:45.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:46.672 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:46.476+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:46.672 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:46.476+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:46.672 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:46.476+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:46.672 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:46.247+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:46.672 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:46.247+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:46.673 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:46.247+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:46.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:46.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:46.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:47.642 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:47.227+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:47.642 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:47.227+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:47.642 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:47.227+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:47.642 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:47.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:47.642 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:47.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:47.642 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:47.440+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:47.640+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:47.640+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:47.640+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:48.436 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:48.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:48.437 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:48.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:48.437 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:48.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:48.691 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:48.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:48.691 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:48.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:48.691 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:48.435+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:48.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:48.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:48.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:48.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:49.441 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:49.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:49.441 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:49.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:49.441 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:49.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:49.721 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:49.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:49.721 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:49.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:49.721 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:49.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:49.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:49.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:49.719+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:50.457 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:50.088+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9652 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:50.457 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:50.163+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:50.457 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:50.163+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:50.458 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:50.163+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:50.717 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:50.454+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:50.718 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:50.454+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:50.718 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:50.454+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:50.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:50.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:50.716+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:51.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:51.118+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:51.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:51.118+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:51.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:51.118+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:51.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:51.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:51.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:51.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:51.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:51.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:52.132 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:51.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:52.132 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:51.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:52.132 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:51.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:52.462 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:52.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:52.463 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:52.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:52.463 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:52.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:52.461+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:52.461+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:52.461+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:53.137 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:52.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:53.137 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:52.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:53.137 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:52.738+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:53.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:53.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:53.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:53.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:53.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:53.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:53.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:53.502+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:53.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:53.502+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:53.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:53.502+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:53.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:53.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:53.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:54.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:54.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:54.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:54.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:54.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:54.488+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:54.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:54.488+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:54.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:54.488+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:54.726+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:54.726+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:54.726+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:55.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:35:55.090+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9729 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:35:55.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:55.159+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:55.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:55.159+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:55.159+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:55.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:55.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:55.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:55.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:55.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:55.528+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:55.712+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:55.712+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:55.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:55.712+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:56.135+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:56.135+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:56.135+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:56.542+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:56.542+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:56.542+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:56.712+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:56.712+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:56.712+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:57.174+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:57.174+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:57.174+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:57.543+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:57.543+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:57.543+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:57.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:57.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:57.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:58.176+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:58.176+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:58.176+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:58.516+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:58.516+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:58.516+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:58.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:58.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:58.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:59.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:59.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:35:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:35:59.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:35:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:59.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:59.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:35:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:35:59.555+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:35:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:59.615+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:59.615+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:35:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:35:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:35:59.615+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:00.090+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9807 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:00.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:00.206+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:00.206+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:00.206+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:00.594+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:00.594+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:00.594+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:00.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:00.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:00.646+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:01.216+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:01.216+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:01.216+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:01.636+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:01.636+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:01.636+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:01.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:01.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:01.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:02.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:02.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:02.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-16T22:36:02.901 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-02-16T22:36:02.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:36:02.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-02-16T22:36:02.969 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:02.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:02.969 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:02.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:02.969 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:02.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:02.970 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:02.675+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:02.970 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:02.675+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:02.970 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:02.675+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:02.973 INFO:tasks.workunit.client.0.smithi002.stderr:mkdir: cannot create directory ‘/home/ubuntu/cephtest/mnt.0/client.0’: Input/output error 2024-02-16T22:36:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-16T22:36:02.974 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-16T22:36:02.974 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-16T22:36:02.975 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-02-16T22:36:02.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-16T22:36:02.975 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-16T22:36:03.041 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-02-16T22:36:03.041 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-16T22:36:03.566 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:03.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:03.566 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:03.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:03.566 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:03.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:03.685+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:03.685+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:03.685+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:03.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:03.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:03.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:04.196+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:04.196+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:04.196+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:04.670+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:04.670+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:04.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:04.670+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:04.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:04.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:04.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:05.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:05.092+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9885 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:05.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:05.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:05.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:05.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:05.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:05.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:05.707+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:05.707+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:05.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:05.707+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:05.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:05.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:05.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:06.688 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:06.268+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:06.688 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:06.268+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:06.689 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:06.268+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:06.742+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:06.742+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:06.742+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:06.686+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:06.686+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:06.686+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:07.639 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:07.256+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:07.639 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:07.256+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:07.639 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:07.256+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:07.778+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:07.778+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:07.778+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:07.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:07.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:07.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:08.630 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:08.303+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:08.631 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:08.303+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:08.631 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:08.303+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:08.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:08.757+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:08.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:08.757+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:08.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:08.757+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:08.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:08.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:08.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:09.646 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:09.290+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:09.646 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:09.290+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:09.646 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:09.290+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:09.745+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:09.745+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:09.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:09.745+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:09.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:09.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:09.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:10.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:10.095+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 9962 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:10.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:10.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:10.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:10.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:10.780+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:10.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:10.780+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:10.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:10.780+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:10.660+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:10.660+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:10.660+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:11.680 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:11.300+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:11.680 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:11.300+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:11.680 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:11.300+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:11.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:11.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:11.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:11.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:11.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:11.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:11.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:12.665 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:12.259+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:12.665 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:12.259+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:12.665 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:12.259+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:12.939 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:12.814+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:12.939 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:12.814+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:12.939 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:12.814+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:12.939 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:12.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:12.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:12.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:12.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:12.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:13.567 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:13.282+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:13.567 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:13.282+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:13.567 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:13.282+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:13.711+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:13.711+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:13.711+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:13.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:13.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:13.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:13.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:13.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:13.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:14.251+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:14.251+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:14.251+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:15.098 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:14.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:15.099 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:14.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:15.099 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:14.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:15.099 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:14.728+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:15.099 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:14.728+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:15.099 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:14.728+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:15.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:15.097+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10039 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:15.228+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:15.228+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:15.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:15.228+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:16.011 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:15.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:16.011 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:15.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:16.011 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:15.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:16.011 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:15.746+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:16.011 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:15.746+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:16.012 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:15.746+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:16.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:16.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:16.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:17.186 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:16.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:17.186 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:16.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:17.186 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:16.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:17.186 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:16.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:17.187 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:16.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:17.187 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:16.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:17.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:17.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:17.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:17.753+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:17.753+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:17.753+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:17.685+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:17.685+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:17.685+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:18.219+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:18.219+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:18.219+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:18.765+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:18.765+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:18.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:18.765+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:18.715+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:18.715+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:18.715+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:19.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:19.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:19.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:20.100 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:19.761+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:20.101 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:19.761+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:20.101 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:19.761+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:20.101 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:19.746+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:20.101 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:19.746+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:20.102 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:19.746+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:20.098+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10117 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:20.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:20.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:20.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:20.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:20.792+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:20.792+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:20.792+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:20.706+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:20.706+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:20.706+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:21.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:21.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:21.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:21.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:21.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:21.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:21.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:21.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:21.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:21.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:21.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:21.727+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:22.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:22.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:22.181+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:23.188 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:22.820+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:23.188 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:22.820+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:23.188 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:22.820+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:23.188 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:22.743+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:23.189 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:22.743+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:23.189 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:22.743+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:23.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:23.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:23.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:23.787+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:23.787+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:23.787+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:23.699+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:23.699+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:23.699+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:24.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:24.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:24.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:24.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:24.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:24.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:24.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:24.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:24.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:24.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:24.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:24.702+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:25.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:25.100+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10195 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:25.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:25.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:25.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:25.759 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:25.693+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:25.759 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:25.693+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:25.759 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:25.693+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:26.127 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:25.757+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:26.128 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:25.757+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:26.128 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:25.757+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:26.126+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:26.126+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:26.126+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:27.162 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:26.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:27.163 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:26.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:27.163 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:26.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:27.163 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:26.733+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:27.163 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:26.733+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:27.163 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:26.733+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:27.160+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:27.160+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:27.160+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:27.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:27.780+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:27.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:27.780+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:27.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:27.780+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:27.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:27.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:27.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:28.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:28.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:28.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:28.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:28.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:28.808+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:28.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:28.808+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:28.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:28.808+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:28.687+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:28.687+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:28.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:28.687+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:29.110+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:29.110+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:29.110+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:29.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:29.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:29.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:29.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:29.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:29.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:29.691+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:29.691+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:29.691+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:30.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:30.102+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10272 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:30.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:30.086+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:30.086+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:30.086+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:30.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:30.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:30.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:30.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:30.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:30.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:30.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:30.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:30.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:30.707+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:31.370 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:31.092+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:31.371 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:31.092+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:31.371 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:31.092+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:31.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:31.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:31.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:31.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:31.749+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:31.749+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:31.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:31.749+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:32.129+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:32.129+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:32.129+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:33.084 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:32.747+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:33.084 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:32.747+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:33.084 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:32.747+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:33.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:32.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:33.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:32.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:33.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:32.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:33.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:33.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:33.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:33.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:33.707+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:33.707+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:33.707+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:33.778+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:33.778+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:33.778+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:34.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:34.070+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:34.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:34.070+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:34.443 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:34.070+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:34.681+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:34.681+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:34.681+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:35.011 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:34.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:35.011 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:34.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:35.011 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:34.747+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:35.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:35.103+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10349 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:35.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:35.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:35.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:35.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:35.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:35.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:35.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:35.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:35.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:35.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:35.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:35.717+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:35.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:35.717+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:35.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:35.717+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:36.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:36.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:36.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:36.676+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:36.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:36.676+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:36.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:36.676+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:36.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:36.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:36.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:37.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:37.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:37.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:37.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:37.674+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:37.674+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:37.675+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:37.721+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:37.721+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:37.721+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:38.103+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:38.103+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:38.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:38.103+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:38.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:38.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:38.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:38.678+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:38.678+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:38.678+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:39.056+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:39.056+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:39.056+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:39.684+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:39.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:39.684+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:39.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:39.684+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:39.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:39.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:39.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:39.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:40.105+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10427 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:40.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:40.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:40.029+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:40.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:40.673+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:40.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:40.673+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:40.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:40.673+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:40.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:40.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:40.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:40.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:41.018+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:41.018+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:41.018+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:41.720+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:41.720+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:41.720+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:41.591+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:41.591+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:41.591+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:42.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:42.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:42.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:42.712+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:42.712+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:42.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:42.712+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:42.547+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:42.547+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:42.547+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:43.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:43.069+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:43.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:43.069+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:43.069+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:43.724+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:43.724+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:43.724+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:43.506+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:43.506+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:43.506+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:44.038+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:44.038+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:44.038+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:44.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:44.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:44.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:44.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:44.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:44.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:44.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:45.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:45.106+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10505 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:45.315 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:44.999+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:45.315 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:44.999+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:45.315 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:44.999+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:45.922 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:45.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:45.923 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:45.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:45.923 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:45.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:45.923 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:45.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:45.923 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:45.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:45.923 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:45.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:46.189 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:45.970+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:46.189 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:45.970+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:46.189 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:45.970+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:46.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:46.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:46.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:46.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:46.795+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:46.795+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:46.795+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:47.007+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:47.007+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:47.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:47.007+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:47.831 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:47.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:47.831 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:47.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:47.831 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:47.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:47.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:48.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:47.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:48.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:47.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:48.021+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:48.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:48.021+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:48.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:48.021+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:48.861 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:48.560+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:48.861 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:48.560+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:48.862 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:48.560+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:48.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:48.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:48.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:48.972+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:48.972+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:48.972+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:49.861 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:49.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:49.861 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:49.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:49.862 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:49.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:50.120 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:50.108+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10582 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:50.121 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:49.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:50.121 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:49.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:50.121 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:49.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:50.122 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:50.021+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:50.122 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:50.021+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:50.122 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:50.021+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:50.856 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:50.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:50.857 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:50.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:50.857 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:50.473+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:50.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:50.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:50.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:50.980+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:50.980+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:50.980+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:51.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:51.470+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:51.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:51.470+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:51.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:51.470+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:51.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:51.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:52.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:51.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:52.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:51.945+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:52.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:51.945+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:52.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:51.945+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:52.776 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:52.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:52.777 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:52.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:52.777 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:52.480+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:52.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:52.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:53.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:52.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:52.990+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:52.990+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:52.990+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:53.807 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:53.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:53.807 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:53.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:53.807 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:53.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:53.805+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:54.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:53.805+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:54.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:53.805+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:54.030+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:54.030+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:54.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:54.030+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:54.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:54.768+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:54.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:54.768+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:54.768+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:54.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:54.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:54.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:55.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:36:55.111+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10659 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:36:55.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:55.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:55.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:55.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:55.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:55.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:55.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:55.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:55.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:55.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:55.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:55.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:56.061 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:55.795+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:56.061 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:55.795+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:56.061 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:55.795+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:56.059+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:56.059+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:56.059+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:56.765+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:56.765+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:56.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:56.765+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:56.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:56.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:56.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:57.093+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:57.093+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:57.093+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:57.805 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:57.543+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:57.806 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:57.543+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:57.806 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:57.543+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:57.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:58.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:57.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:58.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:57.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:58.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:58.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:58.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:58.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:58.777 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:58.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:58.777 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:58.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:58.778 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:58.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:59.043 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:58.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:59.043 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:58.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:59.043 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:58.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:59.041+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:59.041+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:36:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:36:59.041+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:36:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:59.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:59.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:59.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:59.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:36:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:36:59.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:36:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:59.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:59.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:36:59.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:36:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:36:59.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:00.113+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10737 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:00.016+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:00.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:00.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:00.838+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:00.838+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:00.838+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:00.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:00.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:00.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:01.000+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:01.000+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:01.000+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:01.813+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:01.813+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:01.813+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:01.594+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:01.594+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:01.594+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:02.031+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:02.031+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:02.031+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:02.797+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:02.797+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:02.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:02.797+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:02.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:02.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:02.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:02.600+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:03.004+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:03.004+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:03.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:03.004+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:03.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:03.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:03.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:03.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:03.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:03.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:03.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:03.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:04.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:04.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:04.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:04.856 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:04.601+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:04.856 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:04.601+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:04.856 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:04.601+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:05.116 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:04.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:05.117 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:04.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:05.117 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:04.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:05.117 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:05.008+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:05.117 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:05.008+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:05.117 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:05.008+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:05.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:05.114+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10815 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:05.873 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:05.613+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:05.873 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:05.613+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:05.873 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:05.613+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:05.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:05.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:06.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:05.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:06.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:06.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:06.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:06.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:06.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:06.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:06.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:06.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:06.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:07.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:07.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:07.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:07.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:07.882+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:07.882+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:07.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:07.882+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:07.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:07.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:07.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:08.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:08.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:08.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:08.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:08.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:08.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:08.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:08.839+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:08.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:08.839+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:08.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:08.839+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:08.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:08.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:08.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:08.681+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:09.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:09.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:09.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:09.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:09.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:09.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:09.819+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:09.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:09.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:09.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:10.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:10.116+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10892 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:10.080+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:10.080+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:10.080+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:10.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:10.855+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:10.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:10.855+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:10.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:10.855+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:10.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:10.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:10.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:10.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:11.307 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:11.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:11.307 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:11.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:11.307 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:11.052+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:11.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:11.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:11.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:11.895+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:11.895+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:11.895+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:12.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:12.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:12.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:12.620+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:12.620+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:12.620+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:12.858+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:12.858+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:12.858+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:13.104+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:13.104+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:13.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:13.104+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:13.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:13.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:13.650+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:13.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:13.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:13.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:13.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:13.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:13.829+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:14.379 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:14.073+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:14.379 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:14.073+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:14.379 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:14.073+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:14.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:14.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:14.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:14.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:14.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:14.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:14.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:14.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:14.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:15.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:15.118+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 10969 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:15.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:15.032+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:15.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:15.032+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:15.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:15.032+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:15.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:15.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:15.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:15.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:15.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:15.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:16.060+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:16.060+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:16.060+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:16.749+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:16.749+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:16.749+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:16.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:16.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:16.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:17.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:17.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:17.019+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:17.759+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:17.759+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:17.759+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:17.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:17.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:17.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:18.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:18.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:18.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:18.738+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:18.738+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:18.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:18.738+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:18.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:18.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:18.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:19.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:19.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:19.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:19.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:19.772+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:19.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:19.772+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:19.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:19.772+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:19.691+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:19.691+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:19.691+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:20.120+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11047 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:20.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:20.060+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:20.060+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:20.060+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:20.795 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:20.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:20.795 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:20.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:20.795 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:20.741+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:21.055 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:20.793+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:21.055 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:20.793+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:21.055 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:20.793+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:21.054+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:21.054+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:21.054+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:21.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:21.805+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:21.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:21.805+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:21.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:21.805+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:21.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:21.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:21.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:22.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:22.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:22.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:23.136 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:22.781+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:23.136 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:22.781+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:23.136 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:22.781+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:23.136 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:22.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:23.137 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:22.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:23.137 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:22.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:23.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:23.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:23.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:23.815+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:23.815+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:23.815+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:23.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:23.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:23.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:24.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:24.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:24.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:25.123 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:24.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:25.124 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:24.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:25.124 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:24.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:25.124 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:24.822+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:25.125 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:24.822+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:25.125 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:24.822+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:25.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:25.122+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11125 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:25.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:25.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:25.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:26.137 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:25.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:26.137 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:25.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:26.137 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:25.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:26.138 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:25.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:26.138 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:25.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:26.138 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:25.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:26.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:26.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:26.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:26.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:27.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:26.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:27.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:26.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:27.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:26.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:27.147 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:26.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:27.147 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:26.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:27.148 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:26.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:27.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:27.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:27.145+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:28.135 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:27.787+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:28.135 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:27.787+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:28.135 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:27.787+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:28.136 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:27.839+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:28.136 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:27.839+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:28.136 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:27.839+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:28.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:28.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:28.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:28.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:29.114 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:28.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:29.115 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:28.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:29.115 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:28.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:29.115 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:28.887+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:29.115 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:28.887+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:29.115 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:28.887+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:29.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:29.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:29.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:30.097 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:29.837+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:30.097 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:29.837+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:30.097 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:29.837+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:30.098 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:29.862+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:30.098 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:29.862+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:30.098 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:29.862+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:30.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:30.123+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11202 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:30.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:30.095+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:30.095+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:30.095+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:31.112 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:30.902+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:31.112 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:30.902+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:31.112 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:30.902+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:31.112 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:30.852+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:31.113 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:30.852+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:31.113 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:30.852+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:31.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:31.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:31.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:31.850+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:31.850+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:31.850+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:31.852+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:31.852+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:31.852+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:32.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:32.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:32.077+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:33.082 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:32.861+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:33.082 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:32.861+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:33.082 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:32.861+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:33.082 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:32.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:33.082 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:32.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:33.082 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:32.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:33.080+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:33.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:33.080+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:33.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:33.080+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:34.111 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:33.832+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:34.111 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:33.832+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:34.111 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:33.832+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:34.111 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:33.843+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:34.111 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:33.843+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:34.112 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:33.843+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:34.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:34.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:34.109+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:35.128 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:34.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:35.128 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:34.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:35.128 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:34.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:35.128 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:34.870+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:35.128 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:34.870+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:35.129 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:34.870+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:35.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:35.125+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11279 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:35.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:35.130+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:35.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:35.130+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:35.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:35.130+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:36.125 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:35.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:36.125 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:35.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:36.125 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:35.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:36.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:35.820+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:36.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:35.820+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:36.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:35.820+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:36.123+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:36.123+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:36.123+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:37.092 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:36.762+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:37.092 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:36.762+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:37.092 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:36.762+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:37.092 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:36.854+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:37.092 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:36.854+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:37.092 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:36.854+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:37.090+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:37.090+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:37.090+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:38.114 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:37.746+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:38.114 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:37.746+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:38.114 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:37.746+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:38.114 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:37.874+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:38.115 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:37.874+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:38.115 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:37.874+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:38.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:38.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:38.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:39.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:38.859+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:39.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:38.859+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:39.166 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:38.859+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:39.166 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:38.739+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:39.166 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:38.739+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:39.166 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:38.739+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:39.163+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:39.163+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:39.163+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:40.130 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:39.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:40.130 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:39.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:40.130 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:39.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:40.130 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:39.842+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:40.130 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:39.842+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:40.130 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:39.842+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:40.128+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11357 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:40.209+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:40.209+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:40.209+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:41.199 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:40.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:41.199 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:40.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:41.199 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:40.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:41.199 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:40.756+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:41.200 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:40.756+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:41.200 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:40.756+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:41.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:41.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:41.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:41.778+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:41.778+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:41.778+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:41.751+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:41.751+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:41.751+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:42.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:42.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:42.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:42.224+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:42.786+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:42.786+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:42.786+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:43.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:42.743+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:43.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:42.743+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:43.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:42.743+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:43.715 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:43.268+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:43.715 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:43.268+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:43.715 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:43.268+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:43.784+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:43.784+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:43.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:43.784+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:43.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:43.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:43.713+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:44.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:44.233+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:44.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:44.233+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:44.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:44.233+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:44.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:44.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:44.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:44.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:44.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:44.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:45.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:45.129+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11435 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:45.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:45.201+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:45.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:45.201+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:45.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:45.201+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:45.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:45.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:45.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:45.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:45.821+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:45.821+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:45.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:45.821+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:46.222+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:46.222+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:46.222+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:46.634+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:46.634+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:46.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:46.634+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:46.830+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:46.830+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:46.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:46.830+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:47.177+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:47.177+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:47.177+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:47.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:47.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:47.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:47.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:47.826+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:47.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:47.826+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:47.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:47.826+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:48.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:48.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:48.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:48.874 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:48.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:48.875 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:48.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:48.875 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:48.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:49.180 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:48.872+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:49.180 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:48.872+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:49.180 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:48.872+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:49.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:49.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:49.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:49.921 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:49.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:49.922 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:49.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:49.922 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:49.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:50.225 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:50.131+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11511 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:49.920+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:49.920+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:50.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:49.920+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:50.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:50.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:50.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:50.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:50.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:50.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:50.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:50.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:50.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:50.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:50.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:50.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:51.125+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:51.125+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:51.125+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:51.918 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:51.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:51.919 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:51.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:51.919 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:51.663+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:52.085+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:52.085+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:52.085+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:52.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:51.916+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:52.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:51.916+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:52.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:51.916+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:52.923 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:52.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:52.924 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:52.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:52.924 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:52.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:52.921+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:52.921+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:52.921+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:53.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:53.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:53.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:53.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:53.878+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:53.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:53.878+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:53.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:53.878+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:53.692+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:53.692+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:53.692+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:54.082+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:54.082+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:54.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:54.082+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:54.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:54.873+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:54.873+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:54.873+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:54.651+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:54.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:54.651+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:54.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:54.651+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:55.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:37:55.133+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11589 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:37:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:55.064+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:55.064+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:55.064+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:55.850 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:55.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:55.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:55.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:55.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:55.604+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:56.113 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:55.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:56.113 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:55.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:56.113 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:55.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:56.111+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:56.111+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:56.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:56.111+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:56.885 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:56.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:56.885 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:56.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:56.885 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:56.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:57.136 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:56.884+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:57.136 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:56.884+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:57.136 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:56.884+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:57.135+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:57.135+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:57.135+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:57.874+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:57.874+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:57.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:57.874+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:57.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:57.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:57.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:58.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:58.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:58.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:58.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:58.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:58.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:58.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:58.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:58.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:58.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:59.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:59.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:37:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:37:59.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:37:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:59.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:59.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:59.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:37:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:37:59.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:37:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:59.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:59.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:37:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:37:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:37:59.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:00.459 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:00.135+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11666 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:00.460 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:00.123+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:00.460 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:00.123+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:00.460 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:00.123+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:00.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:00.545+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:00.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:00.545+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:00.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:00.545+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:01.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:00.848+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:01.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:00.848+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:01.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:00.848+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:01.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:01.086+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:01.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:01.086+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:01.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:01.086+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:01.808 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:01.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:01.808 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:01.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:01.808 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:01.533+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:02.095 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:01.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:02.096 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:01.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:02.096 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:01.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:02.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:02.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:02.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:02.761 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:02.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:02.761 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:02.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:02.761 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:02.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:03.145 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:02.759+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:03.145 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:02.759+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:03.145 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:02.759+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:03.144+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:03.144+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:03.144+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:03.765 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:03.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:03.765 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:03.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:03.765 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:03.502+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:04.168 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:03.764+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:04.168 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:03.764+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:04.168 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:03.764+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:04.166+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:04.166+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:04.166+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:04.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:04.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:04.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:04.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:04.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:04.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:04.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:04.723+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:04.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:04.723+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:04.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:04.723+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:05.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:05.137+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11745 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:05.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:05.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:05.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:05.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:05.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:05.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:05.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:05.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:05.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:05.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:05.684+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:05.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:05.684+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:05.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:05.684+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:06.457 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:06.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:06.458 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:06.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:06.458 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:06.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:06.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:06.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:06.456+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:07.115 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:06.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:07.116 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:06.726+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:07.116 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:06.726+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:07.454 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:07.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:07.454 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:07.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:07.455 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:07.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:07.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:07.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:07.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:08.150 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:07.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:08.150 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:07.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:08.150 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:07.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:08.459 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:08.148+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:08.459 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:08.148+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:08.459 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:08.148+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:08.458+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:08.458+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:08.458+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:09.127 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:08.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:09.127 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:08.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:09.127 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:08.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:09.126+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:09.126+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:09.126+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:09.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:09.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:09.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:09.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:09.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:09.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:09.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:09.720+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:09.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:09.720+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:09.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:09.720+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:10.466 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:10.139+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11821 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:10.466 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:10.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:10.466 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:10.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:10.466 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:10.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:10.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:10.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:10.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:10.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:10.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:10.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:10.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:10.706+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:10.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:10.706+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:10.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:10.706+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:11.455 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:11.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:11.455 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:11.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:11.455 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:11.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:11.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:11.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:11.453+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:11.699+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:11.699+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:11.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:11.699+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:12.189+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:12.189+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:12.189+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:12.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:12.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:12.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:12.415+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:12.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:12.415+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:13.169 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:12.745+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:13.170 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:12.745+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:13.170 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:12.745+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:13.168+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:13.168+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:13.168+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:13.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:13.396+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:13.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:13.396+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:13.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:13.396+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:14.205 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:13.735+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:14.206 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:13.735+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:14.206 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:13.735+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:14.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:14.204+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:14.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:14.204+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:14.204+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:14.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:14.370+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:14.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:14.370+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:14.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:14.370+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:15.149 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:14.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:15.149 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:14.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:15.149 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:14.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:15.418 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:15.141+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11899 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:15.418 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:15.216+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:15.418 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:15.216+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:15.418 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:15.216+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:15.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:15.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:15.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:16.201 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:15.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:16.201 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:15.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:16.201 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:15.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:16.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:16.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:16.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:16.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:16.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:16.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:16.199+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:16.199+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:16.199+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:17.156 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:16.746+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:17.157 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:16.746+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:17.157 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:16.746+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:17.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:17.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:17.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:17.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:17.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:17.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:17.483+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:17.483+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:17.483+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:18.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:18.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:18.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:18.752 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:18.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:18.752 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:18.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:18.752 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:18.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:19.183 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:18.749+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:19.183 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:18.749+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:19.184 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:18.749+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:19.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:19.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:19.180+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:19.786 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:19.504+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:19.786 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:19.504+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:19.786 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:19.504+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:20.145 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:19.783+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:20.145 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:19.783+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:20.145 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:19.783+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:20.142+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 11976 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:20.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:20.196+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:20.196+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:20.196+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:20.835 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:20.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:20.835 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:20.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:20.835 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:20.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:20.833+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:20.833+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:20.833+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:21.531 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:21.234+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:21.531 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:21.234+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:21.531 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:21.234+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:21.812 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:21.528+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:21.812 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:21.528+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:21.812 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:21.528+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:22.222 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:21.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:22.222 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:21.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:22.222 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:21.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:22.220+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:22.220+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:22.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:22.220+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:22.824 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:22.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:22.824 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:22.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:22.824 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:22.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:22.824 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:22.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:22.825 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:22.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:22.825 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:22.572+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:23.193+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:23.193+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:23.193+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:23.866 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:23.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:23.866 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:23.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:23.866 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:23.541+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:24.157 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:23.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:24.157 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:23.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:24.157 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:23.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:24.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:24.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:24.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:24.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:24.867 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:24.512+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:24.867 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:24.512+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:24.867 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:24.512+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:25.138 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:24.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:25.138 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:24.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:25.139 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:24.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:25.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:25.145+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12055 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:25.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:25.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:25.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:25.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:25.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:25.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:25.911 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:25.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:25.911 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:25.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:25.912 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:25.478+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:26.186 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:25.910+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:26.186 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:25.910+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:26.186 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:25.910+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:26.185+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:26.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:26.185+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:26.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:26.185+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:26.866 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:26.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:26.866 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:26.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:26.867 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:26.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:27.152 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:26.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:27.153 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:26.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:27.153 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:26.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:27.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:27.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:27.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:27.826 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:27.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:27.826 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:27.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:27.826 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:27.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:28.184 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:27.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:28.184 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:27.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:28.184 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:27.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:28.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:28.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:28.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:28.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:28.848 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:28.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:28.848 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:28.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:28.848 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:28.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:29.186 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:28.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:29.186 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:28.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:29.186 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:28.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:29.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:29.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:29.184+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:29.889 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:29.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:29.889 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:29.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:29.889 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:29.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:30.148 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:29.887+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:30.149 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:29.887+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:30.149 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:29.887+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:30.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:30.147+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12131 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:30.204+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:30.205+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:30.205+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:30.855 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:30.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:30.855 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:30.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:30.855 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:30.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:31.175 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:30.853+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:31.175 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:30.853+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:31.176 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:30.853+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:31.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:31.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:31.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:31.830 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:31.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:31.830 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:31.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:31.830 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:31.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:32.135 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:31.828+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:32.135 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:31.828+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:32.135 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:31.828+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:32.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:32.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:32.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:32.133+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:32.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:32.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:32.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:32.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:32.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:32.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:33.130 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:32.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:33.131 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:32.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:33.131 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:32.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:33.128+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:33.128+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:33.128+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:33.839 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:33.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:33.839 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:33.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:33.839 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:33.496+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:34.173 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:33.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:34.173 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:33.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:34.173 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:33.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:34.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:34.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:34.171+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:34.849 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:34.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:34.849 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:34.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:34.849 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:34.527+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:35.150 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:34.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:35.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:34.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:35.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:34.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:35.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:35.149+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12207 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:35.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:35.202+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:35.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:35.202+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:35.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:35.202+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:35.848 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:35.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:35.848 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:35.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:35.848 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:35.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:36.189 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:35.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:36.189 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:35.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:36.189 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:35.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:36.187+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:36.187+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:36.187+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:36.882 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:36.509+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:36.882 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:36.509+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:36.882 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:36.509+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:37.212 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:36.881+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:37.212 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:36.881+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:37.212 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:36.881+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:37.210+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:37.210+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:37.210+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:37.893 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:37.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:37.893 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:37.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:37.894 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:37.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:38.184 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:37.891+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:38.185 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:37.891+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:38.185 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:37.891+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:38.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:38.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:38.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:38.183+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:38.927 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:38.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:38.927 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:38.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:38.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:38.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:39.192 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:38.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:39.193 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:38.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:39.193 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:38.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:39.191+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:39.191+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:39.191+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:39.891 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:39.550+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:39.892 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:39.550+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:39.892 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:39.550+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:40.152 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:39.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:40.153 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:39.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:40.153 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:39.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:40.150+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12278 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:40.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:40.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:40.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:40.842 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:40.515+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:40.842 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:40.515+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:40.842 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:40.515+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:41.188 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:40.841+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:41.188 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:40.841+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:41.188 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:40.841+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:41.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:41.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:41.186+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:41.863 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:41.486+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:41.864 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:41.486+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:41.864 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:41.486+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:42.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:41.862+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:42.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:41.862+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:42.152 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:41.862+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:42.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:42.150+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:42.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:42.150+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:42.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:42.150+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:42.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:42.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:42.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:43.142 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:42.818+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:43.142 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:42.818+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:43.143 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:42.818+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:43.455 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:43.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:43.455 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:43.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:43.455 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:43.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:43.454+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:43.454+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:43.454+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:44.142 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:43.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:44.142 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:43.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:44.142 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:43.804+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:44.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:44.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:44.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:44.761 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:44.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:44.762 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:44.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:44.762 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:44.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:45.154 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:44.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:45.154 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:44.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:45.155 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:44.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:45.449 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:45.152+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12350 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:45.449 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:45.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:45.449 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:45.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:45.449 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:45.170+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:45.447+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:45.447+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:45.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:45.447+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:46.138 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:45.782+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:46.138 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:45.782+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:46.138 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:45.782+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:46.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:46.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:46.136+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:46.741 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:46.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:46.741 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:46.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:46.741 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:46.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:47.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:46.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:47.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:46.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:47.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:46.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:47.459 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:47.150+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:47.459 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:47.150+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:47.459 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:47.150+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:47.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:47.458+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:47.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:47.458+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:47.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:47.458+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:48.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:47.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:48.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:47.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:48.151 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:47.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:48.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:48.149+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:48.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:48.149+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:48.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:48.149+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:48.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:48.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:48.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:48.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:48.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:48.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:49.120 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:48.741+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:49.121 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:48.741+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:49.121 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:48.741+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:49.119+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:49.119+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:49.119+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:49.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:49.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:49.481+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:49.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:49.727+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:49.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:49.727+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:49.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:49.727+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:50.472 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:50.154+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12420 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:50.472 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:50.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:50.472 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:50.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:50.472 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:50.114+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:50.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:50.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:50.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:50.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:50.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:50.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:51.149 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:50.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:51.149 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:50.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:51.149 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:50.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:51.147+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:51.147+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:51.147+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:51.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:51.721+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:51.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:51.721+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:51.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:51.721+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:51.515+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:51.515+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:51.515+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:52.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:52.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:52.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:52.769 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:52.513+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:52.769 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:52.513+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:52.769 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:52.513+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:53.084 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:52.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:53.085 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:52.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:53.085 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:52.767+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:53.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:53.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:53.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:53.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:53.747+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:53.747+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:53.747+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:53.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:53.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:53.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:53.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:53.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:53.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:54.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:54.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:54.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:54.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:54.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:54.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:54.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:54.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:54.797 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:54.524+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:55.108 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:54.795+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:55.109 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:54.795+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:55.109 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:54.795+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:55.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:38:55.156+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12492 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:38:55.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:55.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:55.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:55.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:55.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:55.107+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:55.800 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:55.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:55.801 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:55.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:55.801 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:55.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:56.117 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:55.799+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:56.117 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:55.799+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:56.117 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:55.799+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:56.116+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:56.116+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:56.116+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:56.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:56.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:56.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:56.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:56.751+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:56.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:56.751+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:56.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:56.751+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:57.070+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:57.070+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:57.070+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:57.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:57.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:57.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:57.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:57.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:57.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:57.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:57.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:57.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:58.038+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:58.038+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:58.038+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:58.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:58.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:58.567+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:58.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:58.722+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:58.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:58.722+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:58.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:58.722+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:59.056+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:59.056+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:38:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:38:59.056+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:38:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:59.698+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:59.698+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:38:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:38:59.698+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:38:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:59.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:59.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:38:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:38:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:38:59.569+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:00.158+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12563 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:00.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:00.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:00.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:00.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:00.721+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:00.721+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:00.721+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:00.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:00.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:00.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:00.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:01.080+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:01.080+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:01.080+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:01.899 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:01.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:01.899 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:01.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:01.899 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:01.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:01.899 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:01.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:01.899 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:01.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:01.899 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:01.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:02.104+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:02.104+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:02.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:02.104+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:02.742+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:02.742+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:02.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:02.742+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:02.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:02.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:02.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:03.117+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:03.117+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:03.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:03.117+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:03.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:03.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:03.725+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:03.545+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:03.545+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:03.545+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:04.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:04.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:04.140+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:04.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:04.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:04.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:04.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:04.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:04.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:04.536+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:05.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:05.159+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12635 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:05.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:05.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:05.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:05.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:05.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:05.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:05.726+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:05.726+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:05.726+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:05.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:05.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:05.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:06.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:06.208+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:06.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:06.208+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:06.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:06.208+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:06.699+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:06.699+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:06.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:06.699+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:06.549+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:06.549+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:06.549+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:07.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:07.192+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:07.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:07.192+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:07.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:07.192+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:07.689+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:07.689+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:07.689+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:07.584+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:07.584+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:07.584+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:08.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:08.146+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:08.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:08.146+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:08.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:08.146+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:08.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:08.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:08.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:08.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:08.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:08.683+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:08.586+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:08.587+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:08.587+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:09.172+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:09.172+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:09.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:09.172+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:09.686+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:09.686+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:09.686+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:09.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:09.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:09.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:09.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:10.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:10.161+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12705 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:10.152+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:10.152+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:10.152+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:10.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:10.667+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:10.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:10.667+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:10.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:10.667+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:10.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:10.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:10.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:10.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:11.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:11.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:11.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:11.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:11.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:11.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:11.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:11.710+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:11.710+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:11.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:11.710+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:11.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:11.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:11.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:12.144+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:12.144+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:12.144+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:12.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:12.726+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:12.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:12.726+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:12.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:12.726+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:12.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:12.632+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:12.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:12.632+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:12.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:12.632+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:13.101+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:13.101+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:13.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:13.101+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:13.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:13.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:13.740+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:13.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:13.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:13.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:14.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:14.115+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:14.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:14.115+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:14.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:14.115+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:14.772+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:14.772+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:14.772+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:14.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:14.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:14.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:15.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:15.164+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12777 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:15.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:15.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:15.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:15.083+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:15.759+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:15.759+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:15.759+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:15.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:15.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:15.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:15.704+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:16.078+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:16.078+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:16.078+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:16.761+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:16.761+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:16.761+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:16.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:16.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:16.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:16.670+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:17.085+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:17.085+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:17.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:17.085+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:17.799+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:17.799+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:17.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:17.799+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:17.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:17.651+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:17.651+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:17.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:17.651+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:18.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:18.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:18.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:18.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:18.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:18.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:18.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:18.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:18.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:18.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:18.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:18.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:19.031+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:19.031+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:19.031+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:19.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:19.773+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:19.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:19.773+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:19.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:19.773+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:19.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:19.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:19.641+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:20.292 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:20.166+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12848 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:20.293 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:20.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:20.293 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:20.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:20.293 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:20.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:20.820+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:20.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:20.820+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:20.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:20.820+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:20.602+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:20.602+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:20.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:20.602+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:21.047+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:21.047+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:21.047+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:21.852 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:21.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:21.852 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:21.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:21.852 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:21.557+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:21.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:21.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:21.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:22.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:22.002+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:22.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:22.002+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:22.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:22.002+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:22.839 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:22.542+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:22.839 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:22.542+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:22.839 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:22.542+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:22.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:22.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:22.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:22.956+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:22.956+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:22.956+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:23.564+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:23.564+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:23.564+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:23.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:24.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:23.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:24.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:23.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:23.986+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:23.986+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:23.986+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:24.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:24.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:24.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:24.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:24.851 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:24.540+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:25.170 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:24.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:25.170 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:24.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:25.170 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:24.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:25.171 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:24.967+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:25.171 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:24.967+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:25.171 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:24.967+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:25.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:25.168+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12920 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:25.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:25.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:25.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:25.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:25.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:25.811+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:25.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:25.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:25.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:25.981+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:25.981+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:26.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:25.981+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:26.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:26.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:26.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:26.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:26.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:26.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:26.531+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:26.984+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:26.984+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:26.984+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:27.815 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:27.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:27.815 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:27.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:27.815 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:27.521+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:27.813+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:27.813+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:27.813+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:28.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:27.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:28.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:27.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:28.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:27.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:28.838 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:28.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:28.839 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:28.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:28.839 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:28.525+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:28.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:28.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:28.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:29.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:29.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:29.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:29.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:29.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:29.803 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:29.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:29.804 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:29.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:29.804 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:29.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:30.164 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:29.802+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:30.164 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:29.802+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:30.164 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:29.802+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:30.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:29.976+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:30.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:29.976+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:30.165 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:29.976+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:30.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:30.170+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 12984 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:30.771 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:30.506+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:30.771 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:30.506+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:30.772 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:30.506+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:30.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:30.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:30.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:31.002+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:31.002+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:31.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:31.002+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:31.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:31.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:31.522+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:31.746+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:31.746+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:31.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:31.746+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:32.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:32.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:32.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:32.514+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:32.514+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:32.514+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:32.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:32.719+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:32.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:32.719+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:32.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:32.719+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:33.091+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:33.091+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:33.091+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:33.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:33.731+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:33.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:33.731+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:33.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:33.731+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:33.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:33.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:33.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:34.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:34.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:34.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:34.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:34.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:34.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:34.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:34.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:34.744+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:34.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:34.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:34.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:35.298 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:35.172+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13057 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:35.299 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:35.008+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:35.299 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:35.008+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:35.299 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:35.008+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:35.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:35.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:35.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:35.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:35.546+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:35.993 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:35.734+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:35.993 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:35.735+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:35.993 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:35.735+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:35.991+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:35.991+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:35.991+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:36.688+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:36.688+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:36.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:36.688+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:36.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:36.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:36.555+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:37.008+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:37.008+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:37.008+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:37.579+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:37.579+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:37.579+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:37.712+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:37.712+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:37.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:37.712+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:38.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:38.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:38.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:38.004+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:38.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:38.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:38.580+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:38.684+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:38.684+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:38.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:38.684+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:39.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:39.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:39.043+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:39.670+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:39.670+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:39.670+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:39.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:39.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:39.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:40.301 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:40.174+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13128 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:40.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:40.005+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:40.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:40.005+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:40.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:40.005+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:40.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:40.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:40.964 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:40.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:40.964 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:40.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:40.964 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:40.633+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:40.964 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:40.633+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:40.964 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:40.633+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:40.962+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:40.962+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:40.962+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:41.972 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:41.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:41.972 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:41.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:41.972 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:41.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:41.973 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:41.622+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:41.973 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:41.622+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:41.973 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:41.622+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:42.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:41.970+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:42.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:41.970+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:42.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:41.970+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:42.925 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:42.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:42.925 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:42.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:42.926 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:42.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:42.926 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:42.577+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:42.926 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:42.577+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:42.926 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:42.577+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:43.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:42.924+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:43.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:42.924+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:43.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:42.924+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:43.908 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:43.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:43.909 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:43.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:43.909 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:43.658+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:43.909 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:43.595+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:43.909 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:43.595+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:43.909 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:43.595+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:43.907+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:43.907+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:43.907+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:44.863 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:44.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:44.864 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:44.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:44.864 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:44.668+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:44.864 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:44.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:44.864 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:44.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:44.864 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:44.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:45.178 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:44.861+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:45.178 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:44.861+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:45.179 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:44.861+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:45.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:45.177+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13201 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:45.874 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:45.577+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:45.874 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:45.577+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:45.875 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:45.577+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:45.875 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:45.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:45.875 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:45.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:45.875 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:45.644+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:45.873+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:45.873+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:45.873+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:46.844 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:46.564+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:46.844 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:46.564+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:46.845 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:46.564+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:46.845 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:46.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:46.845 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:46.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:46.845 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:46.665+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:46.843+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:46.843+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:46.843+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:47.858 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:47.558+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:47.859 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:47.558+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:47.859 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:47.558+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:47.859 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:47.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:47.859 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:47.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:47.859 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:47.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:47.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:47.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:47.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:48.848 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:48.589+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:48.849 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:48.589+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:48.849 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:48.589+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:48.849 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:48.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:48.849 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:48.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:48.849 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:48.666+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:48.846+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:48.846+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:48.846+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:49.609+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:49.609+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:49.609+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:49.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:49.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:49.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:49.692+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:49.692+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:49.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:49.692+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:50.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:50.177+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13271 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:50.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:50.639+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:50.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:50.639+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:50.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:50.639+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:50.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:50.765+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:50.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:50.765+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:50.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:50.765+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:50.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:50.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:50.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:50.723+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:51.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:51.640+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:51.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:51.640+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:51.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:51.640+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:51.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:51.721+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:51.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:51.721+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:51.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:51.721+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:51.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:51.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:51.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:51.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:51.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:51.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:52.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:52.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:52.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:52.615+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:52.615+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:52.615+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:52.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:52.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:52.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:52.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:52.672+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:53.709+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:53.709+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:53.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:53.709+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:53.592+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:53.592+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:53.592+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:53.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:53.635+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:53.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:53.635+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:53.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:53.635+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:54.720+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:54.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:54.720+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:54.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:54.720+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:54.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:54.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:54.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:54.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:54.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:54.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:54.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:54.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:54.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:54.978 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:54.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:55.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:39:55.180+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13344 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:39:55.737 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:55.619+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:55.737 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:55.619+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:55.737 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:55.619+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:55.737 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:55.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:55.737 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:55.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:55.737 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:55.611+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:55.735+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:55.735+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:55.735+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:56.646+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:56.646+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:56.646+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:56.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:56.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:56.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:56.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:56.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:56.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:56.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:57.639+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:57.639+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:57.639+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:57.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:57.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:57.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:57.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:57.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:57.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:57.659+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:57.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:57.659+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:57.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:57.659+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:58.598+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:58.598+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:58.598+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:58.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:58.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:58.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:58.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:58.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:58.705+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:58.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:58.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:58.652+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:59.610+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:59.610+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:39:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:39:59.610+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:39:59.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:59.713+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:59.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:59.713+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:59.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:39:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:39:59.713+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:39:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:59.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:59.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:39:59.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:39:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:39:59.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:00.182+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13415 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:00.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:00.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:00.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:00.658+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:00.658+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:00.658+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:00.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:00.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:00.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:01.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:01.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:01.624+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:01.651+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:01.651+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:01.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:01.651+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:01.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:01.677+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:01.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:01.677+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:01.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:01.677+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:02.634+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:02.634+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:02.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:02.634+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:02.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:02.691+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:02.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:02.691+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:02.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:02.691+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:02.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:02.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:02.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:03.656+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:03.656+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:03.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:03.656+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:03.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:03.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:03.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:03.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:03.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:03.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:03.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:03.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:03.695+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:04.692+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:04.692+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:04.692+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:04.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:04.702+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:04.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:04.702+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:04.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:04.702+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:04.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:04.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:04.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:04.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:05.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:05.184+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13488 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:05.645+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:05.645+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:05.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:05.645+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:05.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:05.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:05.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:05.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:05.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:05.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:05.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:05.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:05.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:06.681+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:06.681+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:06.681+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:06.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:06.695+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:06.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:06.695+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:06.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:06.695+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:06.639+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:06.639+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:06.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:06.639+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:07.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:07.708+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:07.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:07.708+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:07.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:07.708+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:07.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:07.731+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:07.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:07.731+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:07.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:07.731+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:07.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:07.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:07.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:07.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:07.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:07.609+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:08.726+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:08.726+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:08.726+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:08.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:08.771+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:08.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:08.771+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:08.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:08.771+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:08.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:08.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:08.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:08.589+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:09.809 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:09.542+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:09.809 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:09.542+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:09.810 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:09.542+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:09.810 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:09.701+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:09.810 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:09.701+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:09.810 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:09.701+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:10.168 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:09.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:10.168 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:09.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:10.168 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:09.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:10.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:10.187+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13558 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:10.853 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:10.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:10.854 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:10.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:10.854 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:10.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:10.854 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:10.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:10.854 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:10.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:10.854 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:10.495+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:10.852+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:10.852+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:10.852+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:11.863 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:11.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:11.864 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:11.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:11.864 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:11.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:11.864 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:11.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:11.864 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:11.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:11.865 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:11.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:11.862+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:11.862+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:11.862+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:12.769 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:12.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:12.770 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:12.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:12.770 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:12.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:12.892+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:12.892+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:13.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:12.892+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:13.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:12.768+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:13.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:12.768+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:13.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:12.768+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:13.814 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:13.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:13.814 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:13.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:13.814 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:13.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:14.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:13.856+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:14.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:13.856+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:14.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:13.856+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:14.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:13.813+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:14.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:13.813+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:14.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:13.813+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:14.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:14.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:14.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:14.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:14.797 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:14.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:15.191 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:14.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:15.191 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:14.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:15.191 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:14.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:15.191 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:14.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:15.191 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:14.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:15.192 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:14.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:15.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:15.189+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13631 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:15.803 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:15.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:15.804 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:15.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:15.804 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:15.526+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:15.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:15.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:15.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:16.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:15.914+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:16.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:15.914+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:16.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:15.914+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:16.940 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:16.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:16.941 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:16.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:16.941 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:16.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:16.941 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:16.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:16.941 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:16.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:16.941 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:16.544+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:16.939+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:16.939+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:16.939+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:17.930 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:17.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:17.930 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:17.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:17.930 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:17.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:17.930 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:17.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:17.930 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:17.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:17.930 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:17.568+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:17.928+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:17.928+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:17.928+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:18.929 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:18.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:18.929 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:18.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:18.929 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:18.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:18.929 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:18.617+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:18.929 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:18.617+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:18.930 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:18.617+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:18.927+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:19.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:18.927+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:19.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:18.927+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:19.956 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:19.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:19.956 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:19.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:19.957 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:19.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:19.957 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:19.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:19.957 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:19.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:19.957 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:19.657+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:20.213 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:20.191+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13702 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:20.214 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:19.955+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:20.214 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:19.955+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:20.214 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:19.955+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:20.965 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:20.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:20.965 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:20.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:20.965 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:20.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:20.966 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:20.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:20.966 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:20.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:20.966 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:20.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:20.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:20.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:20.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:21.713+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:21.713+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:21.713+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:21.597+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:21.597+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:21.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:21.597+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:22.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:22.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:22.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:22.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:22.743+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:22.743+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:22.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:22.743+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:22.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:22.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:22.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:22.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:22.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:22.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:23.025+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:23.025+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:23.025+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:23.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:23.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:23.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:23.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:23.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:23.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:23.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:23.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:23.633+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:24.028+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:24.028+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:24.028+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:24.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:24.778+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:24.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:24.778+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:24.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:24.778+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:24.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:24.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:24.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:24.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:25.302 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:25.193+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13775 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:25.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:24.992+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:25.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:24.992+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:25.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:24.992+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:25.957 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:25.768+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:25.958 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:25.768+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:25.958 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:25.768+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:25.958 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:25.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:25.958 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:25.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:25.958 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:25.653+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:25.956+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:25.956+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:25.956+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:26.765+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:26.765+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:26.765+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:26.686+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:26.686+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:26.686+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:27.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:27.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:27.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:27.744+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:27.744+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:27.744+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:27.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:27.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:27.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:28.024+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:28.024+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:28.024+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:28.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:28.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:28.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:28.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:28.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:28.637+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:28.986+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:28.986+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:28.986+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:29.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:29.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:29.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:29.725+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:29.597+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:29.597+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:29.597+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:30.322 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:30.195+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13845 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:30.322 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:30.035+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:30.323 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:30.035+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:30.323 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:30.035+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:30.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:30.578+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:30.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:30.578+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:30.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:30.578+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:30.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:30.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:30.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:30.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:30.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:30.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:31.041+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:31.041+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:31.041+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:31.613+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:31.613+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:31.613+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:31.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:31.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:31.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:31.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:32.091+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:32.091+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:32.091+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:32.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:32.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:32.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:32.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:32.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:32.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:33.137+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:33.137+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:33.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:33.137+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:33.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:33.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:33.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:33.647+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:33.647+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:33.647+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:34.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:34.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:34.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:34.858+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:34.858+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:34.858+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:34.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:34.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:34.671+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:35.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:35.196+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13918 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:35.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:35.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:35.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:35.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:35.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:35.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:35.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:35.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:35.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:35.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:35.842+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:35.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:35.842+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:35.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:35.842+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:36.149+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:36.149+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:36.149+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:36.849+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:36.849+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:36.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:36.849+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:36.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:36.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:36.683+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:37.160+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:37.160+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:37.160+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:37.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:37.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:37.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:37.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:37.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:37.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:37.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:37.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:37.682+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:38.178+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:38.178+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:38.178+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:38.836+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:38.836+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:38.836+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:38.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:38.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:38.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:39.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:39.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:39.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:39.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:39.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:39.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:39.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:39.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:39.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:39.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:39.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:39.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:40.199+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 13989 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:40.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:40.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:40.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:40.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:40.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:40.669+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:40.854+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:40.854+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:40.854+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:41.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:41.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:41.115+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:41.870+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:41.870+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:41.870+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:41.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:41.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:41.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:41.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:42.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:42.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:42.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:42.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:42.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:42.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:42.660+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:42.660+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:42.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:42.660+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:43.184+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:43.184+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:43.184+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:43.854+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:43.854+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:43.854+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:43.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:43.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:43.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:43.710+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:44.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:44.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:44.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:45.148 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:44.880+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:45.149 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:44.880+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:45.149 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:44.880+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:45.149 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:44.734+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:45.149 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:44.734+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:45.149 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:44.734+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:45.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:45.202+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14062 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:45.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:45.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:45.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:45.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:45.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:45.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:45.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:45.918+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:45.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:45.918+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:45.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:45.918+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:45.718+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:45.718+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:45.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:45.718+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:46.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:46.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:46.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:46.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:46.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:46.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:47.161 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:46.926+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:47.161 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:46.926+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:47.162 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:46.926+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:47.162 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:46.724+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:47.162 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:46.724+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:47.162 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:46.724+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:47.160+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:47.160+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:47.160+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:47.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:47.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:47.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:47.894+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:47.709+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:47.709+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:47.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:47.709+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:48.137+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:48.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:48.137+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:48.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:48.137+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:48.877+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:48.877+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:48.877+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:48.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:48.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:48.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:48.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:48.977 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:48.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:49.090+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:49.090+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:49.090+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:49.872+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:49.872+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:49.872+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:49.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:49.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:49.688+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:50.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:50.204+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14132 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:50.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:50.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:50.131+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:51.126 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:50.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:51.126 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:50.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:51.126 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:50.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:51.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:50.724+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:51.127 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:50.724+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:51.127 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:50.724+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:51.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:51.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:51.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:51.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:51.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:51.829+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:51.749+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:51.749+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:51.749+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:52.102+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:52.102+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:52.102+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:53.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:52.759+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:53.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:52.759+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:53.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:52.759+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:53.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:52.814+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:53.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:52.814+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:53.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:52.814+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:53.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:53.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:53.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:54.184 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:53.825+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:54.184 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:53.825+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:54.184 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:53.825+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:54.185 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:53.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:54.185 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:53.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:54.185 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:53.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:54.182+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:54.182+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:54.182+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:55.208 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:54.819+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:55.208 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:54.819+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:55.209 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:54.819+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:55.209 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:54.752+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:55.209 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:54.752+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:55.209 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:54.752+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:55.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:40:55.206+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14204 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:40:55.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:55.227+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:55.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:55.227+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:55.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:55.227+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:56.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:55.780+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:56.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:55.780+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:56.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:55.780+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:56.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:55.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:56.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:55.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:56.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:55.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:56.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:56.271+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:56.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:56.271+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:56.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:56.271+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:57.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:56.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:57.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:56.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:56.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:56.804+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:56.805+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:56.805+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:57.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:57.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:57.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:57.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:58.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:57.810+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:58.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:57.810+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:58.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:57.810+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:57.849+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:57.849+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:57.849+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:58.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:58.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:58.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:58.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:58.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:58.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:59.195 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:58.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:59.195 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:58.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:59.195 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:58.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:40:59.195 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:58.803+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:59.195 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:58.803+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:59.196 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:58.803+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:40:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:59.193+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:59.193+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:40:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:40:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:40:59.193+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:00.171 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:59.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:00.172 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:59.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:00.172 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:40:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:40:59.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:00.172 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:59.783+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:00.172 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:59.783+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:00.173 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:40:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:40:59.783+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:00.208+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14276 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:00.238+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:00.238+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:00.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:00.238+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:01.202 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:00.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:01.202 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:00.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:01.202 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:00.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:01.203 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:00.791+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:01.203 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:00.791+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:01.203 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:00.791+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:01.201+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:01.201+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:01.201+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:01.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:01.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:01.682+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:01.817+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:01.817+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:01.817+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:02.210+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:02.210+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:02.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:02.210+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:02.669+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:02.669+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:02.669+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:02.840+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:02.840+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:02.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:02.840+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:03.192+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:03.192+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:03.192+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:03.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:03.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:03.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:03.850+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:03.850+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:03.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:03.850+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:04.699 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:04.230+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:04.700 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:04.230+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:04.700 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:04.230+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:04.698+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:04.698+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:04.698+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:04.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:04.831+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:04.831+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:04.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:04.831+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:05.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:05.211+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14348 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:05.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:05.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:05.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:05.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:05.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:05.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:05.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:05.873+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:05.873+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:05.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:05.873+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:05.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:05.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:05.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:05.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:05.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:05.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:06.198+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:06.198+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:06.198+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:07.174 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:06.888+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:07.174 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:06.888+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:07.174 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:06.888+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:07.174 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:06.741+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:07.174 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:06.741+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:07.174 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:06.741+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:07.172+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:07.172+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:07.172+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:08.189 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:07.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:08.189 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:07.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:08.189 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:07.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:08.189 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:07.905+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:08.190 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:07.905+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:08.190 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:07.905+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:08.188+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:08.188+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:08.188+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:09.202 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:08.734+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:09.202 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:08.734+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:09.202 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:08.734+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:09.203 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:08.889+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:09.203 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:08.889+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:09.203 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:08.889+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:09.201+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:09.201+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:09.201+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:10.172 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:09.929+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:10.172 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:09.929+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:10.173 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:09.929+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:10.173 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:09.746+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:10.173 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:09.746+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:10.173 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:09.746+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:10.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:10.213+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14419 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:10.183+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:10.183+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:10.183+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:11.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:10.901+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:11.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:10.901+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:11.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:10.901+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:11.165 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:10.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:11.165 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:10.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:11.166 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:10.748+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:11.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:11.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:11.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:11.906+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:11.906+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:11.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:11.906+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:11.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:11.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:11.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:11.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:11.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:11.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:12.212+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:12.212+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:12.212+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:12.942+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:12.942+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:12.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:12.942+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:12.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:12.701+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:12.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:12.701+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:12.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:12.701+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:13.169+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:13.169+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:13.169+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:13.913+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:13.913+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:13.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:13.913+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:13.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:13.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:13.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:13.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:13.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:13.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:14.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:14.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:14.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:14.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:14.937+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:14.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:14.937+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:14.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:14.937+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:14.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:14.707+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:14.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:14.707+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:14.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:14.707+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:15.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:15.215+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14491 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:15.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:15.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:15.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:15.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:15.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:15.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:15.927 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:15.665+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:15.928 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:15.665+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:15.928 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:15.665+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:16.215 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:15.926+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:16.215 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:15.926+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:16.215 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:15.926+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:16.214+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:16.214+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:16.214+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:16.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:16.901+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:16.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:16.901+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:16.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:16.901+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:16.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:16.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:16.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:16.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:16.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:16.696+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:17.205+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:17.205+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:17.205+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:18.202 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:17.854+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:18.202 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:17.854+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:18.203 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:17.854+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:18.203 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:17.728+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:18.203 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:17.728+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:18.204 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:17.728+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:18.200+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:18.200+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:18.200+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:19.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:18.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:19.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:18.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:19.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:18.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:19.165 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:18.750+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:19.166 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:18.750+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:19.166 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:18.750+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:19.164+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:19.164+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:19.164+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:20.150 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:19.825+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:20.151 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:19.825+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:20.151 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:19.825+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:20.151 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:19.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:20.151 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:19.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:20.151 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:19.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:20.243 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:20.217+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14563 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:20.243 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:20.149+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:20.243 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:20.149+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:20.244 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:20.149+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:21.176 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:20.810+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:21.176 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:20.810+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:21.176 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:20.810+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:21.177 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:20.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:21.177 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:20.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:21.177 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:20.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:21.173+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:21.173+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:21.173+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:21.781+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:21.781+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:21.781+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:21.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:21.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:21.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:22.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:22.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:22.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:22.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:23.164 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:22.824+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:23.164 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:22.824+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:23.164 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:22.824+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:23.165 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:22.782+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:23.165 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:22.782+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:23.165 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:22.782+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:23.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:23.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:23.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:24.196 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:23.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:24.196 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:23.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:24.197 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:23.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:24.197 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:23.851+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:24.197 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:23.851+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:24.197 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:23.851+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:24.194+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:24.194+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:24.194+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:25.174 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:24.810+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:25.175 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:24.810+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:25.175 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:24.810+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:25.175 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:24.822+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:25.175 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:24.822+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:25.175 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:24.822+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:25.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:25.219+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14635 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:25.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:25.171+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:25.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:25.171+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:25.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:25.171+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:26.207 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:25.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:26.207 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:25.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:26.207 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:25.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:26.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:25.847+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:26.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:25.847+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:26.208 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:25.847+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:26.205+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:26.205+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:26.205+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:26.806+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:26.806+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:26.806+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:27.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:26.880+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:27.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:26.880+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:27.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:26.880+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:27.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:27.244+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:27.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:27.244+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:27.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:27.244+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:27.764+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:27.764+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:27.764+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:28.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:27.878+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:28.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:27.878+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:28.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:27.878+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:28.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:28.240+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:28.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:28.240+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:28.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:28.240+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:29.209 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:28.814+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:29.210 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:28.814+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:29.210 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:28.814+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:29.210 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:28.889+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:29.210 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:28.889+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:29.210 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:28.889+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:29.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:29.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:29.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:30.168 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:29.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:30.168 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:29.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:30.168 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:29.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:30.168 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:29.854+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:30.168 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:29.854+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:30.169 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:29.854+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:30.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:30.221+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14706 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:30.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:30.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:30.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:30.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:31.193 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:30.827+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:31.193 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:30.827+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:31.193 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:30.827+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:31.193 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:30.828+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:31.194 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:30.828+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:31.194 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:30.828+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:31.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:31.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:31.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:31.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:31.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:31.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:31.827+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:31.827+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:31.827+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:32.426 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:32.152+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:32.427 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:32.152+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:32.427 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:32.152+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:33.141 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:32.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:33.142 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:32.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:33.142 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:32.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:33.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:32.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:33.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:32.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:33.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:32.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:33.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:33.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:33.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:34.156 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:33.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:34.156 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:33.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:34.157 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:33.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:34.157 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:33.823+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:34.157 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:33.823+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:34.157 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:33.823+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:34.155+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:34.155+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:34.155+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:35.180 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:34.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:35.180 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:34.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:35.180 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:34.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:35.180 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:34.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:35.180 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:34.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:35.180 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:34.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:35.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:35.224+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14774 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:35.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:35.178+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:35.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:35.178+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:35.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:35.178+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:36.145 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:35.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:36.145 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:35.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:36.145 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:35.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:36.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:35.761+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:36.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:35.761+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:36.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:35.761+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:36.143+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:36.143+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:36.143+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:36.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:36.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:36.772+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:36.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:36.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:36.720+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:37.192+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:37.192+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:37.192+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:38.200 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:37.769+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:38.200 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:37.769+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:38.200 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:37.769+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:38.200 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:37.761+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:38.201 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:37.761+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:38.201 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:37.761+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:38.198+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:38.198+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:38.198+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:38.805+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:38.805+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:38.805+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:39.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:38.782+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:39.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:38.782+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:39.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:38.782+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:39.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:39.233+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:39.736 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:39.233+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:39.736 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:39.233+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:40.174 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:39.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:40.174 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:39.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:40.174 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:39.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:40.175 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:39.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:40.175 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:39.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:40.175 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:39.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:40.226+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14846 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:40.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:40.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:40.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:40.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:40.845+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:40.845+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:40.845+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:41.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:40.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:41.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:40.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:41.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:40.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:41.643 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:41.269+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:41.643 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:41.269+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:41.643 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:41.269+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:41.791+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:41.791+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:41.791+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:41.798+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:41.798+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:41.798+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:42.301+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:42.301+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:42.301+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:43.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:42.785+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:43.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:42.785+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:43.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:42.785+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:43.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:42.751+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:43.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:42.751+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:43.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:42.751+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:43.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:43.291+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:43.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:43.291+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:43.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:43.291+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:44.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:43.771+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:44.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:43.771+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:44.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:43.771+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:44.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:43.750+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:44.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:43.750+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:44.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:43.750+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:44.714 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:44.317+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:44.715 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:44.317+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:44.715 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:44.317+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:44.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:44.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:44.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:44.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:44.794+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:44.794+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:44.794+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:45.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:45.229+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14918 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:45.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:45.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:45.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:45.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:45.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:45.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:45.751+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:45.751+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:46.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:45.751+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:45.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:45.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:45.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:46.613 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:46.302+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:46.614 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:46.302+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:46.614 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:46.302+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:46.866 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:46.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:46.866 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:46.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:46.866 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:46.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:47.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:46.864+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:47.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:46.864+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:47.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:46.864+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:47.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:47.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:47.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:47.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:47.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:47.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:48.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:47.875+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:48.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:47.875+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:48.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:47.875+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:47.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:47.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:47.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:48.714 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:48.263+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:48.714 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:48.263+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:48.714 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:48.263+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:48.845+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:48.845+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:48.845+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:48.712+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:48.712+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:48.712+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:49.691 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:49.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:49.692 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:49.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:49.692 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:49.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:49.846+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:49.846+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:49.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:49.846+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:49.690+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:49.690+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:49.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:49.690+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:50.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:50.231+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 14989 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:50.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:50.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:50.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:50.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:51.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:50.862+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:51.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:50.862+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:51.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:50.862+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:50.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:50.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:50.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:51.706 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:51.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:51.706 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:51.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:51.706 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:51.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:51.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:51.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:51.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:51.912+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:51.912+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:51.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:51.912+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:52.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:52.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:52.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:52.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:52.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:52.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:52.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:53.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:52.872+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:53.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:52.872+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:53.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:52.872+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:53.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:53.255+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:53.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:53.255+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:53.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:53.255+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:53.747+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:53.747+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:53.747+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:54.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:53.922+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:54.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:53.922+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:54.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:53.922+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:54.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:54.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:54.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:54.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:55.201 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:54.758+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:55.201 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:54.758+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:55.201 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:54.758+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:55.202 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:54.883+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:55.202 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:54.883+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:55.202 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:54.883+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:55.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:41:55.233+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15061 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:41:55.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:55.199+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:55.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:55.199+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:55.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:55.199+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:56.205 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:55.848+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:56.205 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:55.848+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:56.205 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:55.848+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:56.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:55.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:56.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:55.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:56.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:55.775+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:56.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:56.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:56.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:57.178 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:56.820+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:57.178 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:56.820+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:57.178 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:56.820+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:57.179 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:56.862+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:57.179 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:56.862+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:57.179 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:56.862+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:57.466 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:57.176+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:57.467 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:57.176+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:57.467 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:57.176+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:58.131 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:57.804+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:58.131 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:57.804+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:58.131 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:57.804+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:58.131 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:57.825+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:58.132 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:57.825+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:58.132 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:57.825+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:58.129+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:58.129+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:58.129+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:58.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:58.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:58.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:41:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:58.827+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:58.827+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:58.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:58.827+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:41:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:59.135+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:59.135+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:41:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:41:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:41:59.135+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:00.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:59.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:00.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:59.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:00.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:41:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:41:59.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:00.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:59.782+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:00.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:59.782+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:00.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:41:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:41:59.782+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:00.236+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15133 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:00.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:00.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:00.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:00.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:01.191 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:00.787+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:01.191 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:00.787+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:01.191 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:00.787+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:01.191 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:00.828+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:01.192 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:00.828+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:01.192 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:00.828+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:01.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:01.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:01.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:01.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:01.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:01.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:01.832+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:01.832+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:01.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:01.832+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:02.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:02.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:02.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:03.146 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:02.803+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:03.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:02.803+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:03.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:02.803+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:03.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:02.821+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:03.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:02.821+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:03.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:02.821+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:03.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:03.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:03.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:04.204 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:03.865+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:04.204 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:03.865+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:04.205 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:03.865+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:04.205 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:03.825+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:04.205 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:03.825+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:04.205 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:03.825+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:04.203+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:04.203+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:04.203+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:05.220 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:04.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:05.221 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:04.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:05.221 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:04.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:05.221 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:04.894+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:05.221 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:04.894+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:05.221 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:04.894+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:05.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:05.238+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15205 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:05.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:05.219+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:05.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:05.219+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:05.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:05.219+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:06.177 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:05.882+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:06.177 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:05.882+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:06.177 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:05.882+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:06.178 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:05.849+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:06.178 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:05.849+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:06.178 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:05.849+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:06.175+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:06.175+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:06.175+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:07.131 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:06.834+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:07.131 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:06.834+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:07.132 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:06.834+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:07.132 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:06.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:07.132 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:06.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:07.132 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:06.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:07.130+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:07.130+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:07.130+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:08.141 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:07.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:08.141 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:07.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:08.142 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:07.832+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:08.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:07.814+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:08.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:07.814+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:08.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:07.814+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:08.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:08.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:08.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:09.146 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:08.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:09.146 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:08.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:09.146 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:08.815+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:09.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:08.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:09.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:08.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:09.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:08.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:09.143+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:09.143+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:09.143+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:10.126 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:09.808+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:10.127 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:09.808+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:10.127 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:09.808+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:10.127 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:09.835+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:10.127 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:09.835+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:10.128 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:09.835+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:10.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:10.239+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15276 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:10.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:10.125+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:10.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:10.125+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:10.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:10.125+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:11.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:10.770+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:11.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:10.770+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:11.105 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:10.770+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:11.105 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:10.835+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:11.106 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:10.835+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:11.106 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:10.835+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:11.103+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:11.103+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:11.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:11.103+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:11.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:11.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:11.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:11.860+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:11.860+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:11.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:11.860+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:12.363 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:12.061+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:12.363 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:12.061+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:12.363 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:12.061+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:13.097 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:12.853+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:13.097 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:12.853+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:13.097 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:12.853+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:13.098 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:12.763+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:13.098 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:12.763+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:13.098 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:12.763+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:13.094+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:13.094+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:13.094+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:14.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:13.879+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:14.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:13.879+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:14.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:13.879+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:14.142 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:13.759+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:14.142 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:13.759+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:14.143 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:13.759+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:14.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:14.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:14.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:15.169 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:14.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:15.169 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:14.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:15.169 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:14.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:15.169 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:14.905+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:15.170 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:14.905+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:15.170 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:14.905+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:15.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:15.241+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15348 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:15.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:15.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:15.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:15.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:15.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:15.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:16.132 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:15.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:16.132 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:15.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:16.132 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:15.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:16.132 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:15.936+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:16.133 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:15.936+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:16.133 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:15.936+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:16.130+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:16.130+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:16.130+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:17.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:16.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:17.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:16.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:17.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:16.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:17.164 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:16.959+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:17.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:16.959+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:17.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:16.959+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:17.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:17.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:17.162+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:18.162 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:17.759+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:18.162 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:17.759+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:18.163 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:17.759+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:18.163 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:17.994+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:18.163 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:17.994+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:18.163 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:17.994+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:18.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:18.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:18.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:19.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:18.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:19.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:18.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:19.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:18.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:19.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:18.958+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:19.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:18.958+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:19.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:18.958+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:19.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:19.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:19.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:20.177 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:19.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:20.177 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:19.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:20.178 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:19.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:20.178 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:19.973+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:20.178 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:19.973+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:20.178 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:19.973+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:20.245+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15420 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:20.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:20.247+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:20.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:20.247+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:20.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:20.247+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:20.833+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:20.833+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:20.833+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:21.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:21.016+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:21.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:21.016+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:21.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:21.016+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:21.579 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:21.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:21.579 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:21.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:21.579 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:21.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:21.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:21.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:21.791+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:22.218+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:22.218+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:22.218+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:22.066+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:22.066+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:22.066+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:23.102 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:22.769+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:23.102 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:22.769+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:23.102 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:22.769+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:23.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:23.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:23.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:23.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:23.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:23.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:23.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:23.211+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:23.477 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:23.211+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:23.477 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:23.211+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:24.062 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:23.780+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:24.062 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:23.780+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:24.062 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:23.780+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:24.232+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:24.232+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:24.232+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:24.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:24.060+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:24.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:24.060+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:24.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:24.060+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:24.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:24.819+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:24.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:24.819+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:24.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:24.819+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:25.373 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:25.247+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15492 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:25.373 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:25.258+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:25.374 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:25.258+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:25.374 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:25.258+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:25.374 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:25.084+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:25.374 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:25.084+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:25.374 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:25.084+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:25.856+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:26.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:25.856+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:26.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:25.856+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:26.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:26.086+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:26.227 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:26.086+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:26.227 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:26.086+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:26.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:26.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:26.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:26.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:26.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:26.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:27.216 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:26.903+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:27.216 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:26.903+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:27.216 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:26.903+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:27.217 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:27.095+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:27.217 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:27.095+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:27.217 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:27.095+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:27.214+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:27.214+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:27.214+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:28.221 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:27.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:28.221 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:27.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:28.222 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:27.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:28.222 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:28.136+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:28.222 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:28.136+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:28.222 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:28.136+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:28.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:28.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:28.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:29.188 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:28.888+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:29.188 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:28.888+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:29.189 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:28.888+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:29.189 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:29.118+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:29.189 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:29.118+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:29.189 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:29.118+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:29.187+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:29.187+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:29.187+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:30.186+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:30.186+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:30.186+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:30.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:30.113+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:30.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:30.113+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:30.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:30.113+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:29.937+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:29.937+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:30.227 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:29.937+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:30.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:30.250+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15563 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:31.112 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:30.942+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:31.112 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:30.942+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:31.113 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:30.942+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:31.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:31.110+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:31.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:31.110+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:31.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:31.110+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:31.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:31.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:31.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:31.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:31.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:31.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:31.971+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:31.971+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:31.971+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:32.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:32.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:32.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:32.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:32.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:32.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:32.244+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:32.244+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:32.244+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:33.224 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:32.972+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:33.224 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:32.972+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:33.224 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:32.972+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:33.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:33.125+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:33.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:33.125+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:33.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:33.125+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:33.222+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:33.222+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:33.222+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:34.133 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:33.978+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:34.133 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:33.978+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:34.133 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:33.978+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:34.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:34.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:34.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:34.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:34.132+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:34.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:34.132+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:34.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:34.132+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:35.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:34.951+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:35.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:34.951+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:35.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:34.951+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:35.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:35.133+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:35.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:35.133+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:35.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:35.133+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:35.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:35.252+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15635 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:35.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:35.241+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:35.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:35.241+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:35.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:35.241+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:35.911+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:36.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:35.911+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:36.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:35.911+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:36.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:36.143+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:36.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:36.143+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:36.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:36.143+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:36.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:36.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:36.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:37.160 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:36.898+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:37.160 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:36.898+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:37.160 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:36.898+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:37.247+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:37.247+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:37.247+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:37.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:37.158+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:37.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:37.158+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:37.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:37.158+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:38.154 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:37.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:38.154 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:37.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:38.154 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:37.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:38.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:38.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:38.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:38.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:38.152+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:38.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:38.152+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:38.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:38.152+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:39.109 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:38.837+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:39.109 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:38.837+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:39.110 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:38.837+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:39.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:39.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:39.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:39.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:39.108+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:39.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:39.108+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:39.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:39.108+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:40.117 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:39.850+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:40.117 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:39.850+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:40.117 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:39.850+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:40.379 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:40.255+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15707 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:40.379 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:40.321+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:40.379 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:40.321+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:40.379 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:40.321+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:40.380 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:40.115+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:40.380 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:40.115+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:40.380 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:42:40.115+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:01.369586+0000 (oldest deadline 2024-02-16T22:32:21.369586+0000) 2024-02-16T22:42:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:40.803+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:40.803+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:42:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:42:40.803+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:13.527230+0000 (oldest deadline 2024-02-16T22:32:33.527230+0000) 2024-02-16T22:42:41.711 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:41.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:41.711 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:41.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:41.711 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:41.330+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:42.338+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:42.338+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:42:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:42:42.338+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:32:09.234631+0000 (oldest deadline 2024-02-16T22:32:29.234631+0000) 2024-02-16T22:42:45.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:45.257+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15779 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:50.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:50.260+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15850 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:42:55.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:42:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:42:55.261+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15922 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:00.264+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 15994 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:05.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:05.267+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16066 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:05.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:05.374+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:05.375+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:05.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:05.375+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:05.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:05.451+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:05.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:05.451+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:05.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:05.451+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:06.364 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:06.083+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:06.364 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:06.083+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:06.364 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:06.083+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:06.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:06.449+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:06.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:06.449+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:06.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:06.449+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:06.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:06.362+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:06.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:06.362+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:06.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:06.362+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:07.342 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:07.055+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:07.342 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:07.055+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:07.342 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:07.055+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:07.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:07.340+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:07.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:07.340+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:07.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:07.340+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:07.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:07.460+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:07.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:07.460+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:07.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:07.460+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:08.345+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:08.345+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:08.345+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:08.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:08.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:08.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:08.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:08.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:08.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:08.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:08.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:08.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:09.392 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:09.125+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:09.392 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:09.125+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:09.392 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:09.125+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:09.728 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:09.390+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:09.728 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:09.390+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:09.728 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:09.390+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:09.728 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:09.444+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:09.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:09.444+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:09.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:09.444+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:10.392 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:10.270+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16137 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:10.393 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:10.105+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:10.393 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:10.105+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:10.393 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:10.105+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:10.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:10.417+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:10.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:10.417+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:10.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:10.417+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:10.402+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:10.402+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:10.402+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:11.435 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:11.154+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:11.436 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:11.154+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:11.436 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:11.154+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:11.714 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:11.434+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:11.714 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:11.434+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:11.714 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:11.434+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:11.714 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:11.435+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:11.714 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:11.435+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:11.715 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:11.435+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:12.429+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:12.429+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:12.429+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:12.407+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:12.407+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:12.407+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:12.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:12.201+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:12.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:12.201+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:12.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:12.201+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:13.439 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:13.403+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:13.440 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:13.403+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:13.440 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:13.403+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:13.440 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:13.172+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:13.440 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:13.172+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:13.440 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:13.172+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:13.438+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:13.438+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:13.438+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:14.417+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:14.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:14.417+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:14.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:14.417+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:14.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:14.439+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:14.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:14.439+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:14.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:14.439+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:14.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:14.211+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:14.478 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:14.211+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:14.478 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:14.211+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:15.472 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:15.272+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16209 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:15.472 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:15.373+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:15.472 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:15.373+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:15.472 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:15.373+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:15.473 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:15.216+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:15.473 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:15.216+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:15.473 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:15.216+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:15.470+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:15.470+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:15.470+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:16.325+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:16.325+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:16.325+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:16.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:16.204+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:16.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:16.204+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:16.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:16.204+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:16.517+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:16.517+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:16.517+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:17.516 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:17.315+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:17.516 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:17.315+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:17.516 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:17.315+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:17.517 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:17.229+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:17.517 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:17.229+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:17.517 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:17.229+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:17.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:17.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:17.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:18.516 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:18.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:18.516 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:18.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:18.516 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:18.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:18.516 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:18.262+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:18.517 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:18.262+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:18.517 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:18.262+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:18.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:18.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:18.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:19.536 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:19.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:19.536 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:19.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:19.536 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:19.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:19.537 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:19.273+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:19.537 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:19.273+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:19.537 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:19.273+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:19.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:19.535+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:19.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:19.535+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:19.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:19.535+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:20.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:20.274+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16281 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:20.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:20.322+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:20.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:20.322+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:20.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:20.322+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:20.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:20.236+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:20.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:20.236+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:20.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:20.236+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:20.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:20.545+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:20.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:20.545+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:20.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:20.545+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:21.578 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:21.253+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:21.578 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:21.253+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:21.579 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:21.253+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:21.579 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:21.304+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:21.579 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:21.304+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:21.579 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:21.304+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:21.577+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:21.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:21.577+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:21.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:21.577+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:22.306+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:22.306+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:22.306+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:22.215+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:22.215+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:22.215+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:22.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:22.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:22.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:23.314 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:23.185+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:23.315 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:23.185+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:23.315 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:23.185+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:23.490+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:23.490+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:23.490+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:23.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:23.313+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:23.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:23.313+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:23.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:23.313+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:24.461 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:24.307+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:24.462 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:24.307+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:24.462 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:24.307+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:24.462 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:24.146+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:24.462 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:24.146+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:24.462 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:24.146+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:24.460+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:24.460+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:24.460+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:25.402 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:25.277+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16353 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:25.402 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:25.282+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:25.403 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:25.282+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:25.403 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:25.282+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:25.403 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:25.119+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:25.403 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:25.119+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:25.403 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:25.119+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:25.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:25.477+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:25.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:25.477+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:25.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:25.477+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:26.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:26.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:26.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:26.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:26.168+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:26.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:26.168+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:26.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:26.168+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:26.516+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:26.516+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:26.516+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:27.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:27.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:27.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:27.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:27.138+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:27.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:27.138+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:27.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:27.138+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:27.487+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:27.487+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:27.487+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:28.246+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:28.246+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:28.246+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:28.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:28.110+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:28.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:28.110+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:28.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:28.110+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:28.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:28.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:28.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:29.281+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:29.281+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:29.281+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:29.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:29.114+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:29.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:29.114+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:29.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:29.114+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:29.525+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:29.525+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:29.525+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:30.403 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:30.279+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16424 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:30.404 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:30.080+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:30.404 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:30.080+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:30.404 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:30.080+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:30.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:30.304+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:30.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:30.304+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:30.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:30.304+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:30.511+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:30.511+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:30.511+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:31.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:31.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:31.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:31.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:31.053+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:31.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:31.053+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:31.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:31.053+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:31.529+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:31.529+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:31.529+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:32.299+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:32.299+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:32.299+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:32.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:32.087+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:32.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:32.087+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:32.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:32.087+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:32.518+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:32.518+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:32.518+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:33.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:33.056+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:33.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:33.056+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:33.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:33.056+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:33.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:33.303+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:33.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:33.303+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:33.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:33.303+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:33.537+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:33.537+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:33.537+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:34.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:34.026+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:34.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:34.026+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:34.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:34.026+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:34.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:34.270+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:34.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:34.270+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:34.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:34.270+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:34.516+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:34.516+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:34.516+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:35.284 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:35.246+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:35.284 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:35.246+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:35.284 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:35.246+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:35.285 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:35.000+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:35.285 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:35.000+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:35.285 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:35.000+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:35.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:35.282+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16496 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:35.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:35.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:35.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:35.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:36.470 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:36.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:36.471 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:36.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:36.471 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:36.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:36.471 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:36.027+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:36.471 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:36.027+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:36.471 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:36.027+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:36.469+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:36.469+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:36.469+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:37.469 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:37.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:37.470 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:37.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:37.470 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:37.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:37.470 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:37.037+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:37.470 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:37.037+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:37.470 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:37.037+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:37.468+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:37.468+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:37.468+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:38.428 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:38.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:38.428 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:38.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:38.428 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:38.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:38.429 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:38.002+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:38.429 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:38.002+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:38.429 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:38.002+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:38.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:38.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:38.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:38.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:38.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:38.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:39.450 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:39.231+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:39.450 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:39.231+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:39.450 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:39.231+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:39.450 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:39.016+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:39.451 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:39.016+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:39.451 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:39.016+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:39.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:39.449+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:39.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:39.449+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:39.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:39.449+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:40.308 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:40.284+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16568 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:40.308 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:40.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:40.308 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:40.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:40.308 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:40.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:40.309 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:40.048+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:40.309 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:40.048+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:40.309 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:40.048+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:40.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:40.432+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:40.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:40.432+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:40.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:40.432+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:41.454 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:41.224+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:41.454 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:41.224+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:41.455 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:41.224+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:41.455 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:41.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:41.455 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:41.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:41.455 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:41.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:41.716 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:41.453+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:41.716 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:41.453+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:41.717 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:41.453+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:42.420 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:42.089+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:42.420 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:42.089+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:42.420 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:42.089+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:42.421 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:42.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:42.421 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:42.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:42.421 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:42.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:42.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:42.417+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:42.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:42.417+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:42.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:42.417+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:43.439 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:43.044+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:43.440 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:43.044+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:43.440 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:43.044+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:43.440 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:43.208+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:43.440 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:43.208+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:43.440 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:43.208+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:43.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:43.438+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:43.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:43.438+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:43.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:43.438+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:44.183+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:44.183+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:44.183+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:44.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:44.063+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:44.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:44.063+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:44.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:44.063+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:44.484+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:44.484+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:44.484+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:45.414 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:45.286+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16640 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:45.414 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:45.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:45.414 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:45.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:45.414 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:45.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:45.415 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:45.106+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:45.415 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:45.106+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:45.415 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:45.106+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:45.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:45.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:45.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:45.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:45.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:45.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:46.428 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:46.180+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:46.428 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:46.180+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:46.429 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:46.180+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:46.429 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:46.109+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:46.429 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:46.109+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:46.429 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:46.109+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:46.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:46.426+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:46.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:46.426+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:46.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:46.426+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:47.398 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:47.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:47.398 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:47.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:47.398 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:47.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:47.399 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:47.130+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:47.399 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:47.130+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:47.399 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:47.130+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:47.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:47.396+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:47.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:47.396+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:47.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:47.396+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:48.350+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:48.350+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:48.350+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:48.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:48.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:48.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:48.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:48.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:48.170+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:48.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:48.106+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:48.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:48.106+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:48.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:48.106+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:49.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:49.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:49.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:49.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:49.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:49.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:49.376 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:49.074+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:49.376 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:49.074+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:49.377 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:49.074+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:49.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:49.374+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:49.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:49.374+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:49.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:49.374+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:50.343 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:50.288+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16710 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:50.344 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:50.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:50.344 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:50.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:50.344 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:50.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:50.344 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:50.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:50.344 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:50.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:50.345 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:50.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:50.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:50.341+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:50.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:50.341+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:50.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:50.341+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:51.320 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:51.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:51.321 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:51.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:51.321 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:51.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:51.321 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:51.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:51.321 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:51.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:51.321 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:51.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:51.716 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:51.319+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:51.716 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:51.319+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:51.717 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:51.319+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:52.307 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:52.248+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:52.308 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:52.248+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:52.308 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:52.248+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:52.308 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:52.024+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:52.308 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:52.024+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:52.308 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:52.024+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:52.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:52.306+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:52.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:52.306+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:52.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:52.306+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:53.236+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:53.236+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:53.236+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:53.283+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:53.283+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:53.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:53.283+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:53.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:53.050+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:53.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:53.050+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:53.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:53.050+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:54.199+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:54.199+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:54.199+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:54.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:54.043+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:54.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:54.043+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:54.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:54.043+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:54.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:54.240+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:54.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:54.240+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:54.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:54.240+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:55.377 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:43:55.292+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16783 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:43:55.377 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:55.216+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:55.378 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:55.216+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:55.378 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:55.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:55.378 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:55.071+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:55.378 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:55.071+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:55.378 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:55.071+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:55.378 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:55.258+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:55.379 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:55.258+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:55.379 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:55.258+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:56.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:56.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:56.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:56.279+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:56.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:56.279+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:56.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:56.279+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:56.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:56.062+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:56.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:56.062+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:56.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:56.062+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:57.224+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:57.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:57.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:57.254+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:57.254+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:57.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:57.254+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:57.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:57.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:57.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:57.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:57.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:57.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:58.261+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:58.261+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:58.261+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:58.261+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:58.261+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:58.261+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:58.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:58.130+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:58.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:58.130+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:58.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:58.130+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:59.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:59.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:43:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:43:59.272+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:43:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:59.309+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:59.309+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:59.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:43:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:43:59.309+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:43:59.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:59.177+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:59.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:59.177+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:43:59.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:43:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:43:59.177+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:00.294+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16854 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:00.231+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:00.231+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:00.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:00.231+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:00.350+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:00.350+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:00.350+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:00.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:00.176+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:00.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:00.176+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:00.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:00.176+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:01.235+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:01.235+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:01.235+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:01.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:01.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:01.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:01.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:01.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:01.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:01.207+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:01.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:01.207+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:01.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:01.207+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:02.331+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:02.331+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:02.331+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:02.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:02.203+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:02.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:02.203+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:02.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:02.203+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:02.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:02.264+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:02.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:02.264+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:02.477 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:02.264+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:03.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:03.168+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:03.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:03.168+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:03.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:03.168+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:03.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:03.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:03.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:03.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:03.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:03.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:03.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:03.378+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:03.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:03.378+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:03.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:03.378+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:04.335+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:04.335+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:04.335+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:04.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:04.306+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:04.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:04.306+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:04.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:04.306+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:04.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:04.212+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:04.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:04.212+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:04.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:04.212+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:05.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:05.297+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16927 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:05.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:05.347+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:05.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:05.347+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:05.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:05.347+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:05.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:05.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:05.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:05.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:05.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:05.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:05.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:05.199+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:05.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:05.199+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:05.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:05.199+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:06.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:06.327+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:06.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:06.327+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:06.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:06.327+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:06.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:06.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:06.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:06.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:06.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:06.319+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:06.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:06.169+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:06.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:06.169+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:06.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:06.169+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:07.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:07.374+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:07.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:07.374+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:07.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:07.374+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:07.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:07.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:07.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:07.305+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:07.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:07.126+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:07.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:07.126+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:07.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:07.126+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:08.401 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:08.289+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:08.402 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:08.289+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:08.402 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:08.289+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:08.402 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:08.094+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:08.402 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:08.094+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:08.402 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:08.094+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:08.400+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:08.400+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:08.400+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:09.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:09.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:09.365+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:09.243+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:09.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:09.243+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:09.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:09.243+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:09.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:09.137+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:09.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:09.137+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:09.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:09.137+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:10.404 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:10.300+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 16997 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:10.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:10.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:10.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:10.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:10.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:10.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:10.405 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:10.110+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:10.405 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:10.110+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:10.405 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:10.110+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:10.402+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:10.402+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:10.402+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:11.430 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:11.198+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:11.430 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:11.198+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:11.430 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:11.198+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:11.430 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:11.075+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:11.430 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:11.075+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:11.431 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:11.075+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:11.718 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:11.428+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:11.719 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:11.428+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:11.719 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:11.428+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:12.473 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:12.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:12.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:12.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:12.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:12.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:12.474 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:12.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:12.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:12.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:12.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:12.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:12.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:12.472+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:12.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:12.472+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:12.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:12.472+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:13.459 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:13.236+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:13.459 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:13.236+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:13.459 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:13.236+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:13.459 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:13.075+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:13.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:13.075+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:13.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:13.075+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:13.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:13.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:13.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:13.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:13.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:13.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:14.200+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:14.200+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:14.200+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:14.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:14.076+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:14.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:14.076+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:14.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:14.076+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:14.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:14.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:14.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:15.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:15.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:15.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:15.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:15.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:15.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:15.304 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:15.049+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:15.304 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:15.049+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:15.304 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:15.049+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:15.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:15.302+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17070 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:15.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:15.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:15.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:16.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:16.000+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:16.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:16.000+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:16.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:16.000+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:16.195+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:16.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:16.195+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:16.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:16.195+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:16.969 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:16.556+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:16.969 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:16.556+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:16.969 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:16.556+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:17.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:17.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:17.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:17.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:16.968+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:17.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:16.968+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:17.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:16.968+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:17.929 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:17.552+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:17.930 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:17.552+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:17.930 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:17.552+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:18.175+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:18.176+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:18.176+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:18.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:17.928+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:18.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:17.928+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:18.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:17.928+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:18.921 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:18.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:18.921 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:18.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:18.921 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:18.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:19.132+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:19.132+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:19.132+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:19.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:18.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:19.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:18.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:19.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:18.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:19.922 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:19.565+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:19.922 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:19.565+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:19.923 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:19.565+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:20.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:20.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:20.138+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:20.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:19.921+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:20.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:19.921+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:20.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:19.921+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:20.567 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:20.305+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17141 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:20.901 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:20.566+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:20.901 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:20.566+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:20.901 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:20.566+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:21.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:20.900+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:21.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:20.900+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:21.165 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:20.900+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:21.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:21.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:21.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:21.872 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:21.584+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:21.872 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:21.584+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:21.873 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:21.584+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:22.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:21.871+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:22.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:21.871+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:22.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:21.871+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:22.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:22.119+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:22.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:22.119+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:22.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:22.119+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:22.867 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:22.579+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:22.868 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:22.579+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:22.868 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:22.579+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:23.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:22.866+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:23.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:22.866+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:23.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:22.866+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:23.078+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:23.078+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:23.078+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:23.878 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:23.542+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:23.878 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:23.542+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:23.878 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:23.542+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:24.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:24.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:24.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:24.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:23.877+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:24.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:23.877+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:24.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:23.877+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:24.546+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:24.546+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:24.546+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:25.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:25.075+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:25.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:25.075+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:25.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:25.075+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:25.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:24.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:25.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:24.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:25.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:24.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:25.594 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:25.307+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17214 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:25.816+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:25.816+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:25.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:25.816+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:25.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:25.593+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:25.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:25.593+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:25.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:25.593+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:26.079+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:26.079+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:26.079+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:26.571+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:26.571+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:26.571+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:26.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:26.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:26.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:27.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:27.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:27.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:27.612+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:27.612+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:27.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:27.612+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:27.808+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:27.808+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:27.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:27.808+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:28.085+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:28.085+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:28.085+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:28.793+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:28.793+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:28.793+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:28.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:28.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:28.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:28.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:28.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:28.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:29.111+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:29.111+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:29.111+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:29.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:29.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:29.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:29.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:29.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:29.838+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:29.838+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:29.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:29.838+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:30.434 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:30.309+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17284 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:30.435 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:30.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:30.435 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:30.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:30.435 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:30.142+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:30.590+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:30.590+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:30.590+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:31.168 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:30.807+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:31.168 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:30.807+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:31.168 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:30.807+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:31.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:31.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:31.166+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:31.629+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:31.629+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:31.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:31.629+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:31.842+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:31.842+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:31.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:31.842+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:32.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:32.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:32.150+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:32.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:32.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:32.664+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:32.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:32.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:32.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:32.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:33.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:33.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:33.174+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:33.672+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:33.672+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:33.672+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:33.812+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:33.812+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:33.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:33.812+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:34.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:34.184+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:34.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:34.184+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:34.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:34.184+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:34.624+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:34.624+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:34.624+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:34.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:34.775+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:34.775+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:34.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:34.775+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:35.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:35.312+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17357 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:35.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:35.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:35.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:35.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:35.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:35.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:35.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:35.580+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:35.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:35.580+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:35.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:35.580+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:35.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:35.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:35.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:35.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:36.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:36.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:36.161+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:36.629+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:36.629+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:36.629+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:36.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:36.785+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:36.785+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:36.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:36.785+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:37.203+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:37.203+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:37.203+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:37.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:37.597+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:37.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:37.597+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:37.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:37.597+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:37.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:37.768+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:37.768+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:37.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:37.768+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:38.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:38.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:38.191+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:38.619+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:38.619+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:38.619+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:38.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:38.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:38.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:38.745+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:39.218+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:39.218+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:39.218+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:39.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:39.571+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:39.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:39.571+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:39.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:39.571+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:39.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:39.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:39.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:39.790+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:40.315+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17428 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:40.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:40.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:40.207+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:40.799+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:40.799+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:40.799+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:40.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:40.618+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:40.618+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:40.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:40.618+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:41.186+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:41.186+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:41.186+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:41.842+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:41.842+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:41.842+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:41.601+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:41.601+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:41.601+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:42.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:42.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:42.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:42.890 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:42.608+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:42.891 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:42.608+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:42.891 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:42.608+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:43.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:42.890+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:43.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:42.890+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:43.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:42.890+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:43.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:43.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:43.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:43.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:43.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:43.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:43.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:43.641+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:44.084 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:43.641+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:44.085 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:43.641+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:44.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:43.877+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:44.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:43.877+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:44.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:43.877+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:44.145+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:44.145+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:44.145+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:44.637+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:44.637+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:44.637+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:44.871+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:44.871+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:44.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:44.871+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:45.440 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:45.318+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17492 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:45.441 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:45.103+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:45.441 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:45.103+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:45.441 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:45.103+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:45.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:45.647+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:45.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:45.647+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:45.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:45.647+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:46.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:46.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:46.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:46.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:45.915+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:45.915+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:46.227 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:45.915+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:46.875+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:46.875+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:46.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:46.875+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:46.636+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:46.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:46.636+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:46.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:46.636+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:47.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:47.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:47.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:47.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:47.867+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:47.867+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:47.867+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:47.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:47.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:47.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:47.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:47.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:48.064+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:48.064+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:48.064+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:48.833+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:48.833+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:48.833+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:48.606+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:48.606+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:48.606+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:49.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:49.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:49.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:49.868 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:49.596+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:49.868 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:49.596+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:49.868 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:49.596+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:50.094+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:50.094+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:50.094+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:50.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:49.866+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:50.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:49.866+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:50.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:49.866+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:50.605 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:50.321+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17553 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:50.861 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:50.604+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:50.861 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:50.604+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:50.861 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:50.604+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:51.114 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:50.859+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:51.114 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:50.859+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:51.114 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:50.859+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:51.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:51.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:51.112+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:51.894 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:51.613+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:51.894 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:51.613+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:51.894 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:51.613+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:52.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:52.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:52.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:52.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:51.892+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:52.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:51.892+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:52.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:51.892+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:52.857 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:52.575+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:52.857 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:52.575+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:52.857 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:52.575+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:53.124 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:52.855+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:53.125 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:52.855+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:53.125 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:52.855+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:53.123+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:53.123+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:53.123+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:53.881 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:53.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:53.882 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:53.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:53.882 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:53.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:54.145 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:53.880+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:54.145 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:53.880+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:54.145 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:53.880+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:54.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:54.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:54.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:54.857 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:54.528+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:54.857 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:54.528+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:54.857 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:54.528+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:55.117 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:54.856+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:55.117 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:54.856+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:55.118 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:54.856+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:55.446 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:44:55.323+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17617 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:44:55.446 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:55.116+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:55.446 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:55.116+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:55.446 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:55.116+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:55.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:55.548+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:55.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:55.548+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:55.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:55.548+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:56.150 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:55.807+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:56.150 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:55.807+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:56.150 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:55.807+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:56.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:56.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:56.148+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:56.816 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:56.544+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:56.816 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:56.544+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:56.816 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:56.544+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:57.133 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:56.815+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:57.133 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:56.815+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:57.133 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:56.815+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:57.132+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:57.132+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:57.132+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:57.847 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:57.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:57.848 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:57.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:57.848 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:57.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:58.107 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:57.846+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:58.107 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:57.846+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:58.108 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:57.846+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:58.105+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:58.105+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:58.105+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:58.871 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:58.478+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:58.872 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:58.478+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:58.872 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:58.478+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:59.125 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:58.870+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:59.125 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:58.870+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:59.126 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:58.870+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:44:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:59.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:59.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:44:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:44:59.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:44:59.889 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:59.482+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:59.890 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:59.482+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:44:59.890 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:44:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:44:59.482+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:00.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:59.888+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:00.146 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:59.888+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:00.147 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:44:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:44:59.888+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:00.435 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:00.326+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17679 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:00.436 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:00.145+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:00.436 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:00.145+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:00.436 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:00.145+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:00.433+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:00.433+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:00.433+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:01.183 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:00.916+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:01.184 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:00.916+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:01.184 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:00.916+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:01.438 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:01.182+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:01.438 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:01.182+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:01.438 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:01.182+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:01.723 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:01.436+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:01.723 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:01.436+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:01.723 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:01.436+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:01.939+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:01.939+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:01.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:01.939+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:02.408 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:02.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:02.408 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:02.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:02.408 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:02.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:02.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:02.406+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:02.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:02.406+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:02.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:02.406+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:03.185+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:03.185+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:03.185+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:03.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:02.959+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:03.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:02.959+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:03.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:02.959+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:03.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:03.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:03.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:03.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:03.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:03.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:04.391 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:04.008+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:04.391 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:04.008+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:04.391 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:04.008+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:04.392 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:04.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:04.392 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:04.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:04.392 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:04.140+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:04.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:04.390+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:04.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:04.390+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:04.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:04.390+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:05.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:05.105+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:05.105+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:05.105+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:05.303 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:05.045+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:05.303 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:05.045+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:05.303 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:05.045+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:05.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:05.329+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17743 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:05.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:05.431+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:05.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:05.431+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:05.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:05.431+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:06.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:06.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:06.139+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:06.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:06.047+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:06.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:06.047+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:06.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:06.047+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:06.475+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:06.475+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:06.475+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:07.449 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:07.158+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:07.450 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:07.158+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:07.450 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:07.158+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:07.450 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:07.082+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:07.450 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:07.082+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:07.450 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:07.082+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:07.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:07.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:07.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:07.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:07.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:07.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:08.438 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:08.173+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:08.438 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:08.173+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:08.438 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:08.173+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:08.438 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:08.034+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:08.439 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:08.034+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:08.439 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:08.034+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:08.437+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:08.437+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:08.437+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:09.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:09.145+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:09.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:09.145+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:09.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:09.145+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:09.417 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:09.023+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:09.418 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:09.023+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:09.418 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:09.023+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:09.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:09.415+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:09.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:09.415+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:09.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:09.415+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:10.333 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:10.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:10.333 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:10.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:10.333 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:10.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:10.334 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:10.070+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:10.334 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:10.070+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:10.334 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:10.070+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:10.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:10.331+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17813 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:10.385+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:10.385+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:10.385+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:11.370 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:11.155+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:11.370 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:11.155+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:11.370 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:11.155+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:11.370 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:11.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:11.371 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:11.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:11.371 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:11.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:11.723 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:11.367+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:11.724 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:11.367+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:11.724 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:11.367+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:12.390 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:12.183+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:12.390 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:12.183+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:12.390 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:12.183+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:12.390 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:12.029+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:12.391 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:12.029+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:12.391 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:12.029+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:12.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:12.389+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:12.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:12.389+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:12.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:12.389+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:13.355 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:13.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:13.355 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:13.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:13.355 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:13.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:13.355 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:13.050+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:13.355 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:13.050+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:13.355 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:13.050+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:13.611 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:13.354+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:13.611 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:13.354+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:13.611 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:13.354+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:14.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:14.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:14.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:14.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:14.404 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:14.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:14.405 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:14.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:14.405 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:14.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:14.405 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:14.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:14.403+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:14.403+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:14.403+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:15.459 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:15.334+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17886 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:15.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:15.129+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:15.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:15.129+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:15.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:15.129+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:15.460 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:15.248+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:15.461 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:15.248+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:15.461 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:15.248+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:15.461 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:15.366+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:15.461 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:15.366+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:15.461 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:15.366+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:16.390 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:16.264+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:16.390 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:16.264+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:16.391 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:16.264+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:16.391 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:16.126+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:16.391 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:16.126+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:16.391 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:16.126+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:16.389+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:16.389+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:16.389+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:17.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:17.172+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:17.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:17.172+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:17.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:17.172+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:17.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:17.230+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:17.477 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:17.230+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:17.477 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:17.230+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:17.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:17.373+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:17.478 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:17.373+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:17.478 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:17.373+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:18.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:18.187+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:18.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:18.187+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:18.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:18.187+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:18.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:18.258+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:18.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:18.258+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:18.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:18.258+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:18.341+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:18.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:18.341+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:18.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:18.341+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:19.235+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:19.235+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:19.235+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:19.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:19.240+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:19.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:19.240+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:19.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:19.240+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:19.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:19.325+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:19.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:19.325+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:19.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:19.325+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:20.462 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:20.336+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 17957 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:20.462 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:20.296+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:20.463 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:20.296+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:20.463 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:20.296+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:20.463 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:20.186+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:20.463 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:20.186+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:20.463 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:20.186+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:20.463 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:20.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:20.464 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:20.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:20.464 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:20.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:21.276+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:21.276+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:21.276+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:21.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:21.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:21.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:21.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:21.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:21.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:21.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:21.164+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:21.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:21.164+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:21.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:21.164+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:22.247+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:22.247+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:22.247+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:22.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:22.280+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:22.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:22.280+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:22.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:22.280+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:22.179+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:22.179+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:22.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:22.179+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:23.274+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:23.274+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:23.274+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:23.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:23.259+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:23.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:23.259+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:23.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:23.259+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:23.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:23.176+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:23.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:23.176+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:23.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:23.176+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:24.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:24.203+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:24.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:24.203+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:24.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:24.203+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:24.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:24.285+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:24.477 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:24.285+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:24.477 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:24.285+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:24.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:24.285+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:24.478 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:24.285+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:24.478 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:24.285+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:25.463 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:25.339+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18030 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:25.464 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:25.271+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:25.464 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:25.271+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:25.464 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:25.271+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:25.464 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:25.311+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:25.465 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:25.311+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:25.465 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:25.311+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:25.465 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:25.156+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:25.465 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:25.156+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:25.465 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:25.156+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:26.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:26.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:26.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:26.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:26.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:26.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:26.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:26.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:26.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:26.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:26.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:26.166+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:26.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:26.166+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:26.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:26.166+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:27.331+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:27.331+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:27.332+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:27.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:27.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:27.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:27.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:27.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:27.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:27.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:27.126+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:27.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:27.126+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:27.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:27.126+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:28.339 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:28.293+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:28.339 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:28.293+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:28.339 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:28.293+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:28.340 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:28.077+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:28.340 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:28.077+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:28.340 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:28.077+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:28.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:28.338+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:28.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:28.338+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:28.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:28.338+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:29.328 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:29.265+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:29.329 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:29.265+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:29.329 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:29.265+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:29.329 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:29.056+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:29.329 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:29.056+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:29.329 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:29.056+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:29.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:29.327+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:29.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:29.327+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:29.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:29.327+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:30.344 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:30.285+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:30.344 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:30.285+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:30.345 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:30.285+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:30.345 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:30.060+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:30.345 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:30.061+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:30.345 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:30.061+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:30.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:30.343+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18100 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:30.351+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:30.351+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:30.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:30.351+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:31.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:31.095+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:31.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:31.095+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:31.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:31.095+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:31.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:31.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:31.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:31.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:31.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:31.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:31.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:31.304+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:31.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:31.304+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:31.477 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:31.304+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:32.335 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:32.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:32.335 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:32.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:32.335 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:32.262+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:32.335 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:32.077+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:32.335 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:32.077+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:32.336 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:32.077+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:32.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:32.333+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:32.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:32.333+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:32.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:32.333+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:33.375 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:33.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:33.375 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:33.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:33.375 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:33.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:33.375 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:33.118+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:33.376 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:33.118+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:33.376 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:33.118+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:33.373+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:33.373+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:33.373+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:34.423 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:34.221+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:34.424 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:34.221+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:34.424 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:34.221+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:34.424 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:34.097+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:34.424 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:34.097+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:34.424 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:34.097+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:34.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:34.422+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:34.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:34.422+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:34.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:34.422+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:35.347 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:35.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:35.347 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:35.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:35.347 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:35.234+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:35.347 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:35.080+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:35.347 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:35.080+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:35.348 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:35.080+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:35.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:35.345+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18173 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:35.418+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:35.418+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:35.419+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:36.453 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:36.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:36.453 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:36.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:36.453 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:36.226+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:36.454 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:36.061+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:36.454 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:36.061+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:36.454 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:36.061+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:36.452+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:36.452+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:36.452+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:37.439 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:37.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:37.439 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:37.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:37.439 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:37.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:37.440 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:37.037+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:37.440 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:37.037+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:37.440 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:37.037+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:37.438+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:37.438+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:37.438+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:38.463 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:38.180+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:38.463 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:38.180+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:38.463 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:38.180+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:38.464 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:38.062+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:38.464 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:38.062+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:38.464 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:38.062+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:38.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:38.462+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:38.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:38.462+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:38.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:38.462+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:39.444 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:39.212+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:39.445 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:39.212+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:39.445 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:39.212+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:39.445 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:39.049+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:39.445 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:39.049+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:39.445 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:39.049+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:39.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:39.443+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:39.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:39.443+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:39.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:39.443+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:40.349 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:40.255+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:40.350 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:40.255+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:40.350 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:40.255+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:40.350 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:40.069+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:40.350 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:40.069+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:40.350 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:40.069+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:40.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:40.348+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18244 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:40.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:40.482+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:40.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:40.482+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:40.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:40.482+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:41.268+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:41.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:41.268+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:41.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:41.268+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:41.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:41.030+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:41.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:41.030+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:41.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:41.030+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:41.506+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:41.506+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:41.506+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:42.310+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:42.310+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:42.310+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:42.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:42.067+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:42.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:42.067+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:42.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:42.067+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:42.512+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:42.512+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:42.512+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:43.337 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:43.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:43.338 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:43.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:43.338 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:43.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:43.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:43.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:43.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:43.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:43.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:43.336+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:43.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:43.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:43.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:43.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:43.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:43.514+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:44.330 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:44.075+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:44.330 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:44.075+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:44.331 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:44.075+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:44.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:44.377+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:44.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:44.377+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:44.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:44.377+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:44.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:44.540+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:44.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:44.540+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:44.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:44.540+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:45.329 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:45.064+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:45.330 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:45.064+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:45.330 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:45.064+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:45.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:45.350+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18317 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:45.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:45.328+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:45.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:45.328+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:45.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:45.328+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:45.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:45.555+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:45.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:45.555+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:45.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:45.555+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:46.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:46.055+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:46.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:46.055+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:46.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:46.055+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:46.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:46.288+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:46.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:46.288+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:46.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:46.288+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:46.573+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:46.573+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:46.573+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:47.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:47.098+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:47.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:47.098+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:47.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:47.098+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:47.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:47.335+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:47.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:47.335+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:47.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:47.335+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:47.592+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:47.592+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:47.592+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:48.344+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:48.344+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:48.344+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:48.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:48.138+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:48.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:48.138+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:48.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:48.138+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:48.610+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:48.610+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:48.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:48.610+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:49.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:49.169+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:49.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:49.169+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:49.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:49.169+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:49.354+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:49.354+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:49.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:49.354+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:49.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:49.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:49.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:50.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:50.353+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18387 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:50.342+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:50.342+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:50.342+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:50.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:50.127+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:50.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:50.127+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:50.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:50.127+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:50.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:50.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:50.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:50.685+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:51.344+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:51.344+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:51.344+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:51.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:51.158+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:51.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:51.158+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:51.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:51.158+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:51.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:51.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:51.718+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:52.295+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:52.295+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:52.295+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:52.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:52.197+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:52.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:52.197+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:52.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:52.197+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:53.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:52.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:53.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:52.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:53.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:52.740+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:53.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:53.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:53.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:53.329+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:53.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:53.162+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:53.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:53.162+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:53.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:53.162+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:53.708+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:53.708+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:53.708+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:54.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:54.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:54.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:54.300+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:54.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:54.180+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:54.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:54.180+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:54.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:54.180+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:54.691+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:54.691+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:54.691+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:55.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:45:55.356+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18460 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:45:55.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:55.348+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:55.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:55.348+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:55.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:55.348+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:55.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:55.156+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:55.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:55.156+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:55.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:55.156+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:55.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:55.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:55.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:55.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:55.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:55.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:56.346+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:56.346+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:56.346+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:56.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:56.181+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:56.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:56.181+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:56.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:56.181+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:56.628+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:56.628+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:56.628+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:57.356+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:57.356+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:57.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:57.356+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:57.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:57.168+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:57.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:57.168+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:57.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:57.168+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:57.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:57.639+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:57.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:57.639+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:57.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:57.639+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:58.340+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:58.340+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:58.340+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:58.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:58.160+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:58.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:58.160+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:58.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:58.160+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:58.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:58.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:58.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:59.360+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:59.360+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:45:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:45:59.360+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:45:59.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:59.134+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:59.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:59.134+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:59.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:45:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:45:59.134+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:45:59.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:59.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:59.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:59.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:45:59.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:45:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:45:59.654+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:00.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:00.359+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18531 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:00.326+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:00.326+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:00.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:00.326+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:00.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:00.147+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:00.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:00.147+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:00.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:00.147+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:00.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:00.633+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:00.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:00.633+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:00.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:00.633+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:01.295+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:01.295+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:01.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:01.295+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:01.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:01.183+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:01.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:01.183+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:01.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:01.183+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:01.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:01.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:01.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:02.666 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:02.233+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:02.666 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:02.233+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:02.667 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:02.233+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:02.667 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:02.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:02.667 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:02.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:02.667 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:02.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:02.665+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:02.665+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:02.665+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:03.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:03.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:03.278+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:03.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:03.187+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:03.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:03.187+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:03.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:03.187+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:03.698+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:03.698+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:03.698+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:04.265+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:04.265+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:04.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:04.265+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:04.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:04.218+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:04.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:04.218+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:04.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:04.218+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:04.710+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:04.710+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:04.710+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:05.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:05.361+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18604 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:05.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:05.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:05.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:05.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:05.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:05.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:05.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:05.199+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:05.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:05.199+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:05.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:05.199+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:05.691+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:05.691+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:05.691+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:06.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:06.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:06.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:06.196+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:06.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:06.170+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:06.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:06.170+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:06.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:06.170+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:06.686+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:06.686+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:06.686+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:07.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:07.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:07.213+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:07.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:07.159+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:07.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:07.159+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:07.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:07.159+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:07.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:07.658+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:07.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:07.658+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:07.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:07.658+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:08.229+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:08.229+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:08.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:08.229+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:08.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:08.182+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:08.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:08.182+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:08.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:08.182+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:08.634+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:08.634+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:08.634+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:09.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:09.138+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:09.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:09.138+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:09.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:09.138+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:09.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:09.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:09.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:09.225+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:09.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:09.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:09.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:09.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:09.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:09.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:10.386 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:10.363+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18674 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:10.387 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:10.133+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:10.387 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:10.133+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:10.387 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:10.133+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:10.387 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:10.188+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:10.387 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:10.188+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:10.387 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:10.188+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:10.615+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:10.615+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:10.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:10.615+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:11.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:11.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:11.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:11.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:11.120+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:11.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:11.120+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:11.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:11.120+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:11.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:11.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:11.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:11.646+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:12.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:12.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:12.144+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:12.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:12.137+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:12.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:12.137+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:12.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:12.137+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:12.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:12.614+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:12.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:12.614+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:12.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:12.614+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:13.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:13.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:13.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:13.165+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:13.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:13.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:13.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:13.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:13.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:13.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:13.569+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:13.569+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:13.569+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:14.128+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:14.128+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:14.128+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:14.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:14.064+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:14.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:14.064+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:14.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:14.064+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:14.540+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:14.540+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:14.540+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:15.369 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:15.177+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:15.369 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:15.177+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:15.369 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:15.177+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:15.369 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:15.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:15.369 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:15.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:15.370 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:15.079+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:15.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:15.367+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18747 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:15.507+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:15.507+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:15.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:15.507+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:16.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:16.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:16.163+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:16.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:16.046+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:16.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:16.046+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:16.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:16.046+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:16.530+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:16.530+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:16.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:16.530+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:17.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:17.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:17.127+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:17.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:17.074+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:17.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:17.074+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:17.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:17.074+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:17.570+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:17.570+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:17.570+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:18.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:18.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:18.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:18.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:18.076+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:18.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:18.076+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:18.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:18.076+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:18.534+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:18.534+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:18.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:18.534+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:19.121+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:19.121+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:19.121+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:19.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:19.118+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:19.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:19.118+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:19.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:19.118+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:19.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:19.510+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:19.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:19.510+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:19.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:19.510+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:20.372 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:20.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:20.372 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:20.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:20.372 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:20.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:20.372 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:20.149+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:20.373 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:20.149+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:20.373 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:20.149+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:20.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:20.370+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18818 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:20.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:20.495+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:20.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:20.495+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:20.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:20.495+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:21.447 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:21.062+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:21.448 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:21.062+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:21.448 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:21.062+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:21.448 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:21.195+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:21.448 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:21.195+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:21.448 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:21.195+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:21.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:21.446+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:21.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:21.446+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:21.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:21.446+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:22.400 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:22.040+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:22.401 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:22.040+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:22.401 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:22.040+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:22.401 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:22.187+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:22.401 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:22.187+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:22.401 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:22.187+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:22.724 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:22.399+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:22.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:22.399+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:22.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:22.399+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:23.450 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:23.049+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:23.450 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:23.049+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:23.450 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:23.049+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:23.450 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:23.211+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:23.450 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:23.211+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:23.450 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:23.211+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:23.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:23.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:23.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:24.459 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:24.078+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:24.460 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:24.078+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:24.460 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:24.078+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:24.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:24.262+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:24.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:24.262+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:24.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:24.262+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:24.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:24.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:24.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:25.374 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:25.216+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:25.375 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:25.216+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:25.375 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:25.216+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:25.375 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:25.120+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:25.375 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:25.120+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:25.375 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:25.120+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:25.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:25.373+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18891 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:25.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:25.408+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:25.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:25.408+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:25.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:25.408+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:26.436 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:26.119+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:26.436 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:26.119+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:26.436 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:26.119+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:26.437 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:26.208+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:26.437 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:26.208+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:26.437 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:26.208+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:26.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:26.435+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:26.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:26.435+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:26.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:26.435+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:27.469 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:27.091+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:27.469 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:27.091+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:27.469 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:27.091+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:27.470 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:27.188+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:27.470 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:27.188+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:27.470 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:27.188+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:27.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:27.468+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:27.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:27.468+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:27.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:27.468+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:28.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:28.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:28.124+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:28.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:28.182+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:28.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:28.182+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:28.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:28.182+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:28.510+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:28.510+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:28.510+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:29.099+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:29.099+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:29.099+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:29.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:29.207+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:29.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:29.207+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:29.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:29.207+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:29.475+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:29.475+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:29.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:29.475+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:30.377 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:30.092+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:30.378 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:30.092+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:30.378 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:30.092+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:30.378 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:30.233+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:30.378 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:30.233+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:30.378 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:30.233+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:30.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:30.376+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 18961 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:30.505+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:30.505+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:30.505+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:31.056+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:31.056+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:31.056+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:31.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:31.238+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:31.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:31.238+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:31.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:31.238+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:31.728 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:31.476+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:31.728 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:31.476+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:31.728 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:31.476+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:32.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:32.197+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:32.460 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:32.197+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:32.461 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:32.197+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:32.461 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:32.062+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:32.461 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:32.062+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:32.461 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:32.062+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:32.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:32.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:32.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:32.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:32.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:32.458+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:33.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:33.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:33.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:33.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:33.216+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:33.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:33.216+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:33.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:33.216+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:33.501+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:33.501+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:33.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:33.501+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:34.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:34.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:34.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:34.070+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:34.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:34.203+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:34.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:34.203+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:34.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:34.203+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:34.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:34.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:34.531+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:35.380 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:35.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:35.380 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:35.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:35.381 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:35.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:35.381 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:35.240+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:35.381 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:35.240+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:35.381 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:35.240+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:35.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:35.379+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19034 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:35.527+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:35.527+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:35.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:35.527+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:36.035+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:36.035+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:36.035+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:36.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:36.199+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:36.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:36.199+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:36.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:36.199+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:36.515+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:36.515+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:36.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:36.515+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:36.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:36.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:36.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:37.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:37.170+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:37.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:37.170+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:37.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:37.170+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:37.961 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:37.509+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:37.962 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:37.509+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:37.962 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:37.509+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:37.960+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:37.960+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:37.960+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:38.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:38.179+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:38.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:38.179+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:38.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:38.179+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:38.926 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:38.551+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:38.926 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:38.551+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:38.926 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:38.551+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:39.205 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:38.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:39.206 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:38.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:39.206 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:38.925+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:39.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:39.204+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:39.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:39.204+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:39.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:39.204+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:39.877 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:39.549+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:39.877 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:39.549+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:39.877 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:39.549+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:40.223 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:39.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:40.223 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:39.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:40.223 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:39.876+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:40.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:40.382+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19105 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:40.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:40.222+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:40.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:40.222+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:40.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:40.222+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:40.842 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:40.590+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:40.842 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:40.590+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:40.842 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:40.590+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:41.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:40.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:41.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:40.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:41.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:40.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:41.550 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:41.231+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:41.550 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:41.231+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:41.550 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:41.232+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:41.841 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:41.549+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:41.841 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:41.549+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:41.841 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:41.549+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:42.218 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:41.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:42.218 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:41.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:42.218 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:41.840+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:42.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:42.217+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:42.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:42.217+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:42.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:42.217+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:42.851 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:42.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:42.851 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:42.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:42.851 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:42.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:42.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:42.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:42.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:43.614 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:43.265+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:43.614 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:43.265+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:43.614 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:43.265+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:43.866 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:43.613+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:43.866 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:43.613+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:43.866 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:43.613+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:43.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:43.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:43.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:44.636 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:44.245+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:44.637 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:44.245+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:44.637 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:44.245+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:44.841+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:44.841+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:44.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:44.841+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:44.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:44.636+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:44.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:44.636+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:44.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:44.636+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:45.509 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:45.385+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19178 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:45.509 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:45.232+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:45.510 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:45.232+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:45.510 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:45.232+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:45.815 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:45.643+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:45.815 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:45.643+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:45.815 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:45.643+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:46.207 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:45.813+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:46.207 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:45.813+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:46.207 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:45.813+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:46.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:46.205+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:46.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:46.205+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:46.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:46.205+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:46.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:46.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:46.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:46.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:46.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:46.775+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:46.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:46.639+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:46.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:46.639+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:46.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:46.639+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:47.635 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:47.229+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:47.636 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:47.229+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:47.636 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:47.229+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:47.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:47.779+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:47.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:47.779+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:47.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:47.779+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:47.632+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:47.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:47.632+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:47.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:47.632+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:48.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:48.184+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:48.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:48.184+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:48.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:48.184+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:48.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:48.742+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:48.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:48.742+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:48.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:48.742+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:48.626+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:48.626+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:48.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:48.626+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:49.669 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:49.231+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:49.670 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:49.231+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:49.670 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:49.231+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:49.668+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:49.668+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:49.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:49.668+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:49.733+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:49.733+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:49.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:49.733+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:50.514 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:50.387+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19248 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:50.514 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:50.230+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:50.515 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:50.230+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:50.515 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:50.230+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:50.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:50.711+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:50.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:50.711+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:50.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:50.711+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:50.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:50.628+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:50.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:50.628+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:50.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:50.628+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:51.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:51.201+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:51.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:51.201+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:51.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:51.201+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:51.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:52.024 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:51.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:52.024 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:51.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:52.025 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:51.715+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:52.025 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:51.715+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:52.025 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:51.715+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:52.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:52.207+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:52.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:52.207+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:52.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:52.207+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:52.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:52.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:52.659+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:52.747+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:52.747+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:52.747+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:53.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:53.169+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:53.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:53.169+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:53.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:53.169+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:53.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:53.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:53.670+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:53.786+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:53.786+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:53.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:53.786+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:54.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:54.142+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:54.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:54.142+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:54.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:54.142+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:54.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:54.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:54.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:54.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:54.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:54.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:54.758+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:54.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:54.758+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:54.977 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:54.758+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:55.475 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:46:55.389+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19320 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:46:55.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:55.184+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:55.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:55.184+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:55.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:55.184+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:55.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:55.584+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:55.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:55.584+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:55.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:55.584+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:55.724+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:55.724+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:55.724+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:56.623 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:56.224+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:56.624 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:56.224+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:56.624 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:56.224+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:56.711+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:56.711+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:56.711+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:56.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:56.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:56.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:56.622+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:57.488 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:57.230+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:57.489 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:57.230+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:57.489 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:57.230+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:57.785 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:57.696+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:57.785 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:57.696+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:57.785 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:57.696+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:57.786 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:57.641+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:57.786 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:57.641+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:57.786 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:57.641+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:58.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:58.271+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:58.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:58.271+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:58.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:58.271+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:58.655+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:58.655+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:58.655+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:58.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:58.702+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:58.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:58.702+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:58.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:58.702+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:59.669 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:59.247+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:59.669 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:59.247+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:59.669 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:46:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:46:59.247+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:46:59.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:59.667+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:59.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:59.667+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:59.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:46:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:46:59.667+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:46:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:59.701+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:59.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:59.701+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:46:59.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:46:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:46:59.701+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:00.641 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:00.393+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19408 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:00.642 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:00.286+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:00.642 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:00.286+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:00.642 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:00.286+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:00.696+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:00.696+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:00.696+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:00.640+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:00.640+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:00.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:00.640+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:01.596 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:01.294+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:01.596 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:01.294+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:01.596 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:01.294+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:01.660+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:01.660+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:01.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:01.660+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:01.594+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:01.594+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:01.594+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:02.600 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:02.248+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:02.600 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:02.248+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:02.600 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:02.248+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:02.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:02.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:02.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:02.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:02.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:02.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:02.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:02.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:02.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:02.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:02.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:02.598+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:03.592 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:03.296+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:03.593 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:03.296+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:03.593 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:03.296+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:03.591+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:03.591+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:03.591+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:03.621+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:03.621+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:03.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:03.621+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:04.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:04.277+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:04.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:04.277+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:04.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:04.277+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:04.635+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:04.635+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:04.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:04.635+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:04.624+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:04.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:04.624+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:04.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:04.624+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:05.520 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:05.396+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19480 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:05.520 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:05.238+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:05.521 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:05.238+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:05.521 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:05.238+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:05.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:05.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:05.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:05.611+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:05.611+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:05.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:05.611+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:06.575 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:06.286+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:06.575 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:06.286+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:06.575 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:06.286+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:06.574+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:06.574+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:06.574+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:06.650+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:06.650+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:06.650+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:07.546 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:07.261+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:07.546 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:07.261+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:07.546 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:07.261+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:07.544+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:07.544+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:07.544+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:07.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:07.688+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:07.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:07.688+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:07.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:07.688+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:08.664 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:08.495+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:08.664 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:08.495+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:08.664 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:08.495+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:08.665 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:08.309+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:08.665 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:08.309+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:08.665 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:08.309+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:08.662+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:08.662+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:08.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:08.662+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:09.700 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:09.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:09.700 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:09.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:09.700 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:09.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:09.701 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:09.334+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:09.701 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:09.334+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:09.701 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:09.334+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:09.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:09.699+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:09.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:09.699+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:09.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:09.699+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:10.723 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:10.399+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19551 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:10.723 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:10.494+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:10.724 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:10.494+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:10.724 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:10.494+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:10.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:10.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:10.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:10.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:10.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:10.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:10.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:10.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:10.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:10.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:10.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:10.722+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:11.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:11.457+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:11.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:11.457+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:11.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:11.457+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:11.316+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:11.316+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:11.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:11.316+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:12.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:11.764+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:12.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:11.764+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:12.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:11.764+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:12.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:12.464+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:12.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:12.464+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:12.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:12.464+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:12.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:12.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:12.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:12.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:13.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:12.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:13.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:12.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:13.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:12.785+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:13.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:13.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:13.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:13.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:13.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:13.374+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:13.374+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:13.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:13.374+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:14.224 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:13.786+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:14.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:13.786+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:14.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:13.786+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:14.468+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:14.468+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:14.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:14.468+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:14.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:14.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:14.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:14.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:14.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:14.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:15.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:14.755+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:15.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:14.755+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:15.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:14.755+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:15.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:15.402+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19623 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:15.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:15.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:15.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:15.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:15.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:15.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:15.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:15.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:15.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:15.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:15.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:15.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:15.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:15.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:16.485+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:16.485+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:16.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:16.485+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:16.368+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:16.368+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:16.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:16.368+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:17.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:16.786+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:17.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:16.786+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:17.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:16.786+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:17.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:17.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:17.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:17.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:17.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:17.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:17.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:17.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:17.513+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:17.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:17.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:17.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:18.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:18.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:18.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:18.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:18.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:18.469+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:18.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:18.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:18.441+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:18.774+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:18.774+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:18.774+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:19.732 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:19.476+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:19.732 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:19.476+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:19.733 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:19.476+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:19.733 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:19.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:19.733 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:19.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:19.733 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:19.484+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:20.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:19.731+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:20.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:19.731+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:20.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:19.731+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:20.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:20.405+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19695 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:20.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:20.482+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:20.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:20.482+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:20.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:20.482+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:20.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:20.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:20.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:20.764+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:20.764+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:20.764+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:21.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:21.467+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:21.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:21.467+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:21.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:21.467+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:21.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:21.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:21.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:21.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:21.803+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:21.803+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:21.803+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:22.634 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:22.482+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:22.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:22.482+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:22.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:22.482+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:22.635 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:22.378+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:22.635 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:22.378+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:22.635 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:22.378+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:22.794+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:22.794+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:22.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:22.794+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:23.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:23.461+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:23.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:23.461+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:23.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:23.461+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:23.390+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:23.390+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:23.390+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:24.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:23.821+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:24.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:23.821+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:24.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:23.821+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:24.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:24.445+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:24.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:24.445+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:24.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:24.445+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:24.384+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:24.384+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:24.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:24.384+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:24.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:24.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:24.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:25.479 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:25.408+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19767 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:25.480 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:25.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:25.480 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:25.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:25.480 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:25.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:25.850 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:25.478+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:25.851 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:25.478+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:25.851 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:25.478+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:25.850+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:25.850+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:25.850+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:26.900 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:26.506+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:26.900 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:26.506+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:26.901 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:26.506+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:26.901 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:26.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:26.901 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:26.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:26.901 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:26.479+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:26.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:26.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:26.899+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:27.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:27.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:27.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:27.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:27.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:27.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:27.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:27.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:27.489+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:27.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:27.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:27.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:28.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:28.472+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:28.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:28.472+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:28.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:28.472+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:28.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:28.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:28.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:28.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:28.896+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:28.896+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:28.896+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:29.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:29.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:29.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:29.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:29.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:29.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:29.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:29.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:29.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:29.850+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:29.850+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:29.850+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:30.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:30.411+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19838 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:30.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:30.454+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:30.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:30.454+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:30.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:30.454+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:30.410+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:30.410+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:30.410+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:31.224 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:30.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:30.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:30.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:31.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:31.432+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:31.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:31.432+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:31.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:31.432+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:31.393+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:31.393+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:31.393+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:32.224 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:31.784+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:31.784+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:31.784+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:32.481+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:32.481+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:32.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:32.481+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:32.377+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:32.377+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:32.377+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:32.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:32.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:32.818+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:33.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:33.474+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:33.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:33.474+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:33.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:33.474+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:33.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:33.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:33.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:34.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:33.806+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:34.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:33.806+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:34.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:33.806+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:34.452+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:34.452+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:34.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:34.452+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:34.320+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:34.320+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:34.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:34.320+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:35.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:34.757+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:35.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:34.757+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:35.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:34.757+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:35.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:35.414+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19909 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:35.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:35.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:35.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:35.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:35.726 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:35.470+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:35.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:35.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:35.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:35.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:35.783+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:35.783+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:35.783+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:36.424+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:36.424+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:36.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:36.424+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:36.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:36.334+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:36.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:36.334+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:36.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:36.334+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:36.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:36.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:36.767+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:37.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:37.456+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:37.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:37.456+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:37.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:37.456+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:37.298+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:37.298+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:37.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:37.298+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:38.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:37.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:38.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:37.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:38.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:37.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:38.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:38.437+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:38.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:38.438+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:38.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:38.438+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:38.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:38.273+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:38.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:38.273+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:38.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:38.273+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:38.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:38.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:39.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:38.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:39.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:39.465+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:39.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:39.465+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:39.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:39.465+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:39.315+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:39.315+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:39.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:39.315+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:40.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:39.835+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:40.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:39.835+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:40.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:39.835+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:40.543 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:40.416+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 19981 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:40.543 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:40.428+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:40.544 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:40.428+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:40.544 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:40.428+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:40.544 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:40.283+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:40.544 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:40.283+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:40.544 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:40.283+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:40.878+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:40.878+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:41.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:40.878+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:41.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:41.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:41.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:41.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:41.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:41.433+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:41.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:41.252+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:41.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:41.252+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:41.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:41.252+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:42.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:41.866+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:42.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:41.866+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:42.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:41.866+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:42.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:42.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:42.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:42.439+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:42.250+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:42.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:42.250+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:42.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:42.250+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:43.205 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:42.905+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:43.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:42.905+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:43.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:42.905+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:43.451+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:43.451+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:43.451+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:43.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:43.204+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:43.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:43.204+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:43.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:43.204+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:44.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:43.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:44.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:43.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:44.206 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:43.891+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:44.474 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:44.205+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:44.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:44.205+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:44.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:44.205+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:44.916 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:44.495+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:44.916 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:44.495+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:44.917 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:44.495+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:45.182 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:44.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:45.182 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:44.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:45.182 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:44.915+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:45.434 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:45.419+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20052 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:45.434 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:45.181+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:45.434 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:45.181+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:45.434 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:45.181+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:45.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:45.535+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:45.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:45.535+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:45.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:45.535+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:45.910+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:45.910+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:45.910+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:46.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:46.145+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:46.145+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:46.145+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:46.885 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:46.578+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:46.885 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:46.578+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:46.885 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:46.578+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:46.883+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:46.883+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:46.883+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:47.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:47.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:47.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:47.100+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:47.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:47.101+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:47.858 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:47.601+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:47.858 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:47.601+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:47.859 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:47.601+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:48.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:47.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:48.147 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:47.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:48.148 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:47.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:48.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:48.146+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:48.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:48.146+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:48.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:48.146+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:48.849 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:48.570+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:48.849 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:48.570+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:48.849 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:48.570+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:49.164 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:48.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:49.165 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:48.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:49.165 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:48.848+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:49.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:49.163+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:49.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:49.163+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:49.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:49.163+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:49.618+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:49.618+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:49.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:49.618+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:49.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:49.822+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:49.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:49.822+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:49.977 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:49.822+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:50.446 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:50.423+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20123 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:50.446 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:50.186+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:50.446 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:50.186+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:50.446 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:50.186+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:50.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:50.636+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:50.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:50.636+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:50.725 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:50.636+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:51.166 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:50.807+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:51.166 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:50.807+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:51.167 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:50.807+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:51.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:51.165+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:51.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:51.165+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:51.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:51.165+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:51.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:51.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:51.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:51.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:51.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:51.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:51.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:51.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:51.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:51.631+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:52.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:52.193+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:52.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:52.193+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:52.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:52.193+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:52.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:52.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:52.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:52.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:52.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:52.588+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:52.588+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:52.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:52.588+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:53.549 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:53.234+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:53.550 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:53.234+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:53.550 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:53.234+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:53.777+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:53.777+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:53.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:53.777+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:53.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:53.548+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:53.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:53.548+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:53.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:53.548+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:54.570 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:54.245+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:54.570 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:54.245+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:54.570 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:54.245+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:54.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:54.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:54.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:54.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:54.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:54.568+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:54.568+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:54.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:54.568+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:55.552 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:47:55.426+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20194 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:47:55.552 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:55.248+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:55.552 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:55.248+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:55.552 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:55.248+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:55.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:55.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:55.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:55.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:55.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:55.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:55.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:55.612+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:55.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:55.612+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:55.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:55.612+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:56.567 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:56.245+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:56.567 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:56.245+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:56.567 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:56.245+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:56.742+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:56.742+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:56.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:56.742+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:56.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:56.565+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:56.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:56.565+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:56.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:56.565+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:57.527 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:57.267+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:57.527 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:57.267+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:57.527 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:57.267+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:57.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:57.759+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:57.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:57.759+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:57.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:57.759+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:57.525+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:57.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:57.525+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:57.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:57.525+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:58.576 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:58.307+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:58.577 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:58.307+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:58.577 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:58.307+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:58.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:58.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:58.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:58.575+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:58.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:58.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:58.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:58.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:58.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:58.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:59.579 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:59.298+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:59.579 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:59.298+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:59.580 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:47:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:47:59.298+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:47:59.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:59.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:59.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:59.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:59.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:47:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:47:59.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:47:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:59.578+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:59.578+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:47:59.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:47:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:47:59.578+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:00.546 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:00.429+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20266 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:00.547 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:00.286+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:00.547 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:00.286+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:00.547 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:00.286+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:00.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:00.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:00.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:00.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:00.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:00.644+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:00.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:00.545+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:00.545+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:00.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:00.545+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:01.643 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:01.509+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:01.643 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:01.509+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:01.644 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:01.509+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:01.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:01.263+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:01.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:01.263+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:01.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:01.263+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:01.974 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:01.642+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:01.642+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:01.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:01.642+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:02.513 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:02.226+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:02.513 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:02.226+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:02.513 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:02.226+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:02.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:02.511+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:02.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:02.512+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:02.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:02.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:02.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:02.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:02.627+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:03.556 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:03.260+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:03.556 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:03.260+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:03.556 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:03.260+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:03.603+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:03.603+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:03.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:03.603+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:03.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:03.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:03.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:03.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:03.554+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:04.599 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:04.298+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:04.600 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:04.298+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:04.600 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:04.298+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:04.645+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:04.645+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:04.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:04.645+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:04.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:04.598+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:04.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:04.598+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:04.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:04.598+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:05.555 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:05.432+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20337 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:05.556 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:05.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:05.556 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:05.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:05.556 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:05.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:05.585+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:05.585+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:05.586+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:05.599+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:05.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:05.599+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:05.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:05.599+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:06.587 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:06.310+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:06.588 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:06.310+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:06.588 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:06.310+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:06.625+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:06.625+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:06.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:06.625+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:06.586+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:06.586+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:06.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:06.586+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:07.660 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:07.549+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:07.661 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:07.549+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:07.661 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:07.549+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:07.661 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:07.344+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:07.662 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:07.344+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:07.662 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:07.344+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:07.658+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:07.658+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:07.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:07.658+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:08.648 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:08.513+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:08.649 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:08.513+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:08.649 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:08.513+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:08.649 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:08.313+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:08.649 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:08.313+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:08.650 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:08.313+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:08.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:08.647+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:08.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:08.647+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:08.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:08.647+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:09.676 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:09.520+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:09.676 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:09.520+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:09.676 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:09.520+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:09.676 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:09.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:09.677 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:09.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:09.677 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:09.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:09.674+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:09.674+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:09.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:09.674+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:10.642 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:10.435+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20408 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:10.643 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:10.486+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:10.643 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:10.486+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:10.643 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:10.486+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:10.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:10.321+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:10.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:10.321+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:10.644 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:10.321+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:10.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:10.641+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:10.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:10.641+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:10.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:10.641+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:11.636 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:11.476+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:11.637 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:11.476+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:11.637 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:11.476+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:11.637 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:11.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:11.637 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:11.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:11.637 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:11.340+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:11.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:11.635+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:11.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:11.635+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:11.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:11.635+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:12.671 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:12.454+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:12.672 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:12.454+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:12.672 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:12.454+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:12.672 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:12.310+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:12.672 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:12.310+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:12.672 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:12.310+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:12.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:12.670+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:12.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:12.670+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:12.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:12.670+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:13.645 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:13.423+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:13.646 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:13.423+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:13.646 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:13.423+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:13.646 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:13.323+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:13.646 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:13.323+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:13.646 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:13.323+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:13.645+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:13.645+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:13.645+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:14.650 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:14.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:14.651 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:14.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:14.651 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:14.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:14.651 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:14.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:14.651 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:14.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:14.652 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:14.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:14.650+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:14.650+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:14.650+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:15.562 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:15.438+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20479 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:15.563 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:15.398+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:15.563 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:15.398+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:15.563 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:15.398+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:15.563 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:15.354+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:15.563 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:15.354+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:15.564 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:15.354+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:15.677+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:15.677+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:15.677+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:16.716 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:16.442+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:16.717 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:16.442+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:16.717 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:16.442+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:16.717 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:16.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:16.717 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:16.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:16.717 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:16.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:16.715+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:16.715+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:16.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:16.715+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:17.316+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:17.316+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:17.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:17.316+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:17.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:17.478+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:17.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:17.478+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:17.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:17.478+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:17.762+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:17.762+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:17.762+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:18.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:18.459+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:18.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:18.459+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:18.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:18.459+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:18.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:18.297+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:18.297+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:18.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:18.297+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:18.752+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:18.752+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:18.752+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:19.311+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:19.311+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:19.311+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:19.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:19.439+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:19.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:19.439+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:19.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:19.439+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:19.727+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:19.727+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:19.727+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:20.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:20.441+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20551 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:20.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:20.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:20.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:20.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:20.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:20.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:20.324+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:20.324+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:20.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:20.324+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:20.774+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:20.774+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:20.774+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:21.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:21.409+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:21.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:21.409+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:21.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:21.409+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:21.280+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:21.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:21.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:21.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:21.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:21.792+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:21.792+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:21.792+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:22.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:22.382+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:22.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:22.382+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:22.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:22.382+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:22.311+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:22.311+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:22.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:22.311+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:22.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:22.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:22.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:23.387+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:23.387+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:23.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:23.387+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:23.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:23.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:23.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:23.281+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:24.074 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:23.801+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:24.075 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:23.801+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:24.075 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:23.801+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:24.342 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:24.317+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:24.342 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:24.317+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:24.342 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:24.317+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:24.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:24.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:24.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:24.340+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:25.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:24.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:25.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:24.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:25.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:24.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:25.569 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:25.443+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20622 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:25.570 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:25.339+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:25.570 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:25.339+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:25.570 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:25.339+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:25.570 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:25.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:25.570 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:25.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:25.570 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:25.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:25.879+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:25.879+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:25.879+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:26.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:26.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:26.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:26.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:26.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:26.299+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:26.404+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:26.404+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:26.404+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:27.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:26.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:27.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:26.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:27.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:26.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:27.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:27.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:27.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:27.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:27.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:27.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:27.428+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:27.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:27.428+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:27.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:27.428+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:27.861+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:27.861+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:27.861+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:28.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:28.294+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:28.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:28.294+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:28.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:28.294+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:28.463+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:28.463+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:28.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:28.463+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:28.860+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:28.860+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:28.860+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:29.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:29.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:29.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:29.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:29.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:29.314+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:29.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:29.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:29.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:29.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:29.459+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:29.888+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:29.888+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:29.888+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:30.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:30.446+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20693 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:30.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:30.322+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:30.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:30.322+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:30.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:30.322+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:30.424+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:30.424+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:30.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:30.424+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:30.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:30.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:30.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:31.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:31.370+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:31.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:31.370+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:31.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:31.370+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:31.405+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:31.405+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:31.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:31.405+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:31.974+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:31.974+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:31.974+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:32.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:32.368+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:32.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:32.368+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:32.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:32.368+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:32.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:32.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:32.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:32.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:32.953+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:32.953+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:32.953+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:33.409+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:33.409+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:33.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:33.409+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:33.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:33.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:33.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:33.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:34.398 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:33.990+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:34.399 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:33.990+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:34.399 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:33.990+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:34.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:34.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:34.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:34.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:34.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:34.397+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:34.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:34.397+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:34.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:34.397+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:35.301 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:34.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:35.301 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:34.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:35.302 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:34.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:35.573 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:35.450+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20764 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:35.574 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:35.407+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:35.574 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:35.407+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:35.574 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:35.407+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:35.574 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:35.429+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:35.574 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:35.429+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:35.574 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:35.429+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:36.412 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:36.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:36.413 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:36.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:36.413 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:36.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:36.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:36.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:36.472+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:36.411+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:36.411+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:36.411+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:37.448 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:36.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:37.448 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:36.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:37.448 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:36.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:37.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:37.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:37.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:37.447+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:37.447+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:37.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:37.447+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:38.421 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:37.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:38.421 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:37.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:38.422 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:37.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:38.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:38.419+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:38.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:38.419+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:38.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:38.419+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:38.550+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:38.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:38.550+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:38.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:38.550+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:39.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:38.951+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:39.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:38.951+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:39.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:38.951+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:39.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:39.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:39.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:39.548+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:39.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:39.449+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:39.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:39.449+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:39.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:39.449+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:40.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:39.927+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:40.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:39.927+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:40.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:39.927+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:40.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:40.453+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20836 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:40.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:40.488+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:40.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:40.488+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:40.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:40.488+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:40.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:40.595+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:40.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:40.595+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:40.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:40.595+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:41.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:40.880+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:41.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:40.880+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:41.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:40.880+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:41.736 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:41.482+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:41.737 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:41.482+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:41.737 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:41.482+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:41.737 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:41.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:41.737 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:41.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:41.737 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:41.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:42.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:41.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:42.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:41.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:42.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:41.854+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:42.812 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:42.487+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:42.812 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:42.487+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:42.812 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:42.487+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:42.812 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:42.529+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:42.813 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:42.529+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:42.813 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:42.529+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:42.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:42.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:42.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:43.811 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:43.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:43.812 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:43.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:43.812 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:43.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:43.812 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:43.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:43.812 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:43.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:43.812 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:43.491+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:43.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:43.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:43.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:44.850 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:44.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:44.850 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:44.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:44.850 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:44.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:44.850 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:44.480+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:44.851 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:44.480+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:44.851 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:44.480+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:45.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:44.848+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:45.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:44.848+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:45.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:44.848+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:45.584 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:45.457+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20907 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:45.584 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:45.519+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:45.585 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:45.519+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:45.585 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:45.519+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:45.585 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:45.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:45.585 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:45.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:45.585 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:45.510+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:45.873+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:45.873+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:45.873+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:46.907 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:46.538+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:46.907 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:46.538+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:46.907 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:46.538+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:46.908 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:46.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:46.908 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:46.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:46.908 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:46.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:47.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:46.906+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:46.906+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:46.906+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:47.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:47.520+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:47.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:47.520+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:47.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:47.521+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:47.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:47.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:47.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:47.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:47.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:47.471+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:47.942+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:47.942+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:47.942+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:48.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:48.490+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:48.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:48.490+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:48.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:48.490+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:48.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:48.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:48.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:48.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:48.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:48.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:49.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:48.969+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:48.969+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:48.969+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:49.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:49.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:49.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:49.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:49.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:49.526+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:49.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:49.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:49.751 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:49.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:49.751 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:49.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:49.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:49.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:49.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:50.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:50.460+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 20978 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:50.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:50.493+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:50.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:50.493+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:50.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:50.493+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:50.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:50.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:50.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:50.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:51.383 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:50.977+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:51.383 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:50.977+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:51.383 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:50.977+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:51.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:51.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:51.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:51.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:51.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:51.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:51.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:51.494+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:51.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:51.494+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:51.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:51.494+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:51.932+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:51.932+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:51.932+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:52.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:52.427+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:52.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:52.427+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:52.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:52.427+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:52.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:52.541+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:52.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:52.541+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:52.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:52.541+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:53.402 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:52.975+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:53.402 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:52.975+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:53.403 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:52.975+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:53.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:53.401+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:53.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:53.401+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:53.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:53.401+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:53.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:53.535+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:53.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:53.535+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:53.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:53.535+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:53.936+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:53.936+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:53.936+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:54.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:54.378+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:54.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:54.378+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:54.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:54.378+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:54.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:54.500+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:54.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:54.500+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:54.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:54.500+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:54.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:54.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:54.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:55.477 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:48:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:48:55.463+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21049 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:48:55.477 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:55.359+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:55.478 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:55.359+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:55.478 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:55.359+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:55.478 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:55.454+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:55.478 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:55.454+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:55.478 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:55.454+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:56.397 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:56.003+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:56.398 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:56.003+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:56.398 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:56.003+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:56.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:56.412+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:56.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:56.412+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:56.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:56.412+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:56.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:56.396+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:56.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:56.396+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:56.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:56.396+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:57.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:56.960+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:56.960+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:56.960+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:57.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:57.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:57.382+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:57.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:57.368+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:57.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:57.368+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:57.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:57.368+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:57.917+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:57.917+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:57.918+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:58.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:58.410+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:58.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:58.410+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:58.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:58.410+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:58.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:58.380+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:58.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:58.380+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:58.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:58.380+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:59.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:58.933+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:59.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:58.933+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:59.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:58.933+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:48:59.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:59.456+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:59.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:59.456+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:59.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:48:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:48:59.456+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:48:59.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:59.335+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:59.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:59.335+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:48:59.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:48:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:48:59.335+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:59.903+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:59.903+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:48:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:48:59.903+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:00.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:00.466+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21121 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:00.497+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:00.497+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:00.497+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:00.351+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:00.351+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:00.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:00.351+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:01.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:00.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:01.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:00.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:01.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:00.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:01.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:01.470+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:01.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:01.470+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:01.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:01.470+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:01.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:01.366+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:01.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:01.366+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:01.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:01.366+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:02.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:01.817+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:02.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:01.817+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:02.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:01.817+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:02.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:02.505+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:02.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:02.505+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:02.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:02.505+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:02.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:02.398+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:02.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:02.398+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:02.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:02.398+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:02.793+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:02.793+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:02.793+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:03.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:03.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:03.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:03.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:03.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:03.481+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:03.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:03.397+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:03.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:03.397+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:03.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:03.397+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:03.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:03.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:03.770+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:04.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:04.472+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:04.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:04.472+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:04.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:04.472+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:04.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:04.369+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:04.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:04.369+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:04.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:04.369+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:04.792+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:04.792+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:04.792+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:05.594 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:05.469+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21192 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:05.595 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:05.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:05.595 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:05.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:05.595 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:05.326+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:05.595 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:05.444+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:05.595 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:05.444+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:05.595 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:05.444+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:05.974 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:05.799+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:05.799+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:05.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:05.799+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:06.323+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:06.323+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:06.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:06.323+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:06.470+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:06.470+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:06.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:06.470+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:07.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:06.838+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:07.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:06.838+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:07.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:06.838+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:07.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:07.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:07.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:07.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:07.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:07.427+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:07.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:07.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:07.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:07.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:07.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:07.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:07.802+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:07.802+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:07.802+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:08.464+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:08.464+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:08.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:08.464+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:08.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:08.359+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:08.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:08.359+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:08.726 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:08.359+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:08.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:08.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:08.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:09.727 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:09.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:09.727 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:09.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:09.727 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:09.448+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:09.727 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:09.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:09.728 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:09.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:09.728 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:09.403+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:10.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:09.734+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:10.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:09.734+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:10.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:09.734+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:10.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:10.471+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21263 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:10.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:10.426+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:10.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:10.426+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:10.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:10.426+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:10.469+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:10.469+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:10.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:10.469+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:10.774+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:10.774+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:10.774+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:11.395+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:11.395+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:11.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:11.395+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:11.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:11.480+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:11.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:11.480+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:11.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:11.480+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:11.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:11.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:11.794+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:12.431+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:12.431+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:12.431+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:12.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:12.521+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:12.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:12.521+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:12.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:12.521+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:12.808+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:12.808+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:12.808+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:13.553 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:13.443+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:13.553 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:13.443+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:13.553 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:13.443+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:13.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:13.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:13.760+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:13.551+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:13.551+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:13.551+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:14.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:14.405+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:14.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:14.405+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:14.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:14.405+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:14.593+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:14.593+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:14.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:14.593+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:15.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:14.756+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:15.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:14.756+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:15.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:14.756+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:15.602 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:15.474+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21334 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:15.602 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:15.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:15.603 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:15.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:15.603 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:15.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:15.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:15.630+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:15.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:15.630+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:15.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:15.630+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:15.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:15.800+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:15.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:15.800+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:15.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:15.800+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:16.680+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:16.680+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:16.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:16.680+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:16.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:16.446+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:16.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:16.446+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:16.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:16.446+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:16.817+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:16.817+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:16.817+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:17.719 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:17.406+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:17.719 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:17.406+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:17.719 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:17.406+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:17.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:17.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:17.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:17.717+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:17.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:17.861+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:17.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:17.861+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:17.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:17.861+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:18.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:18.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:18.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:18.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:18.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:18.423+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:18.749+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:18.749+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:19.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:18.749+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:18.901+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:18.901+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:19.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:18.901+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:19.385+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:19.385+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:19.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:19.385+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:20.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:19.773+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:20.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:19.773+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:20.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:19.773+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:20.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:19.868+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:20.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:19.868+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:20.227 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:19.868+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:20.601 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:20.478+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21406 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:20.602 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:20.342+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:20.602 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:20.342+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:20.602 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:20.342+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:20.870 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:20.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:20.870 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:20.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:20.870 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:20.811+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:20.869+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:20.869+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:20.869+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:21.320+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:21.320+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:21.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:21.320+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:21.886+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:21.886+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:21.886+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:22.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:21.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:22.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:21.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:22.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:21.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:22.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:22.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:22.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:22.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:22.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:22.331+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:22.874+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:22.874+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:22.874+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:23.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:22.798+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:22.798+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:23.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:22.798+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:23.351+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:23.351+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:23.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:23.351+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:23.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:23.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:23.837+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:23.807+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:23.807+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:24.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:23.807+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:24.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:24.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:24.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:24.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:24.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:24.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:24.779+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:24.779+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:24.779+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:25.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:24.803+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:24.803+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:24.803+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:25.483 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:25.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:25.483 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:25.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:25.484 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:25.352+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:25.771 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:25.482+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21477 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:25.771 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:25.731+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:25.771 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:25.731+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:25.771 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:25.731+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:25.769+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:25.769+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:25.769+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:26.318+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:26.318+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:26.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:26.318+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:27.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:26.790+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:27.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:26.790+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:27.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:26.790+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:26.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:26.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:27.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:26.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:27.357+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:27.357+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:27.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:27.357+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:27.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:27.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:27.766+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:27.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:28.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:27.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:28.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:27.788+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:28.355+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:28.355+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:28.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:28.355+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:28.769+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:28.769+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:28.769+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:28.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:28.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:29.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:28.756+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:29.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:29.312+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:29.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:29.312+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:29.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:29.312+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:29.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:29.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:30.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:29.763+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:29.730+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:29.730+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:30.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:29.730+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:30.609 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:30.485+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21548 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:30.609 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:30.356+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:30.610 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:30.356+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:30.610 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:30.356+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:30.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:30.774+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:30.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:30.774+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:30.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:30.774+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:30.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:30.806+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:30.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:30.806+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:30.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:30.806+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:31.360+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:31.360+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:31.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:31.360+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:32.224 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:31.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:31.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:31.760+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:31.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:31.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:31.785+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:32.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:32.327+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:32.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:32.327+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:32.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:32.327+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:32.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:32.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:32.737+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:33.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:32.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:33.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:32.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:33.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:32.816+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:33.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:33.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:33.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:33.364+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:34.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:33.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:34.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:33.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:34.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:33.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:34.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:33.830+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:34.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:33.830+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:34.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:33.830+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:34.716 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:34.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:34.716 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:34.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:34.716 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:34.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:34.715+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:34.715+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:34.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:34.715+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:34.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:34.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:34.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:34.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:34.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:34.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:35.614 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:35.488+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21619 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:35.614 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:35.425+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:35.614 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:35.425+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:35.614 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:35.425+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:35.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:35.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:35.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:35.847+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:35.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:35.679+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:35.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:35.679+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:35.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:35.679+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:36.435+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:36.435+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:36.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:36.435+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:36.651+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:36.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:36.651+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:36.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:36.651+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:37.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:36.820+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:37.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:36.820+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:37.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:36.820+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:37.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:37.389+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:37.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:37.389+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:37.616+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:37.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:37.616+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:37.726 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:37.616+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:37.868+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:37.868+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:37.868+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:38.657 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:38.387+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:38.657 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:38.387+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:38.657 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:38.387+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:38.884+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:38.884+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:38.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:38.884+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:38.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:38.655+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:38.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:38.656+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:38.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:38.656+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:39.662 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:39.354+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:39.662 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:39.354+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:39.662 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:39.354+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:39.841+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:39.841+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:39.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:39.841+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:39.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:39.661+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:39.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:39.661+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:39.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:39.661+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:40.616 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:40.491+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21691 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:40.616 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:40.349+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:40.617 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:40.350+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:40.617 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:40.350+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:40.890 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:40.634+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:40.890 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:40.634+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:40.890 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:40.634+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:41.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:40.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:41.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:40.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:41.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:40.889+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:41.675 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:41.390+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:41.675 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:41.390+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:41.675 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:41.390+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:41.921+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:41.921+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:41.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:41.921+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:41.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:41.674+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:41.674+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:41.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:41.674+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:42.716 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:42.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:42.716 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:42.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:42.716 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:42.438+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:42.891+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:42.891+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:42.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:42.891+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:42.714+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:42.714+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:42.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:42.714+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:43.457+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:43.457+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:43.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:43.457+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:44.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:43.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:43.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:43.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:43.757+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:43.757+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:43.757+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:44.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:44.410+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:44.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:44.410+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:44.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:44.410+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:45.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:44.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:45.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:44.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:45.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:44.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:45.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:44.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:45.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:44.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:45.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:44.753+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:45.618 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:45.494+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21762 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:45.618 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:45.429+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:45.618 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:45.429+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:45.619 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:45.429+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:45.959 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:45.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:45.959 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:45.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:45.959 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:45.732+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:45.958+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:45.958+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:45.958+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:46.704 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:46.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:46.705 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:46.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:46.705 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:46.444+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:46.703+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:46.703+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:46.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:46.703+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:47.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:46.981+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:47.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:46.981+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:47.417 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:46.981+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:47.697 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:47.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:47.697 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:47.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:47.698 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:47.416+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:47.697+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:47.697+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:47.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:47.697+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:48.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:48.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:48.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:48.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:48.376 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:48.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:48.699 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:48.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:48.699 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:48.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:48.699 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:48.375+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:48.964 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:48.698+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:48.964 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:48.698+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:48.965 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:48.698+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:49.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:48.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:48.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:48.963+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:49.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:49.358+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:49.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:49.358+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:49.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:49.358+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:49.992 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:49.739+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:49.992 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:49.739+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:49.992 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:49.739+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:50.403 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:49.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:50.403 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:49.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:50.403 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:49.991+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:50.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:50.497+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21833 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:50.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:50.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:50.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:50.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:50.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:50.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:50.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:50.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:50.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:50.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:50.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:50.761+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:51.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:51.430+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:51.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:51.430+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:51.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:51.430+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:52.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:51.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:51.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:52.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:51.970+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:51.792+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:51.792+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:52.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:51.792+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:52.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:52.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:52.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:52.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:52.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:52.402+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:53.018 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:52.765+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:53.018 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:52.765+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:53.018 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:52.765+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:53.400 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:53.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:53.400 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:53.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:53.400 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:53.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:53.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:53.398+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:53.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:53.398+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:53.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:53.398+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:54.047+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:54.047+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:54.047+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:53.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:53.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:53.799+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:54.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:54.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:54.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:54.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:54.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:54.388+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:54.838+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:54.838+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:54.838+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:55.003+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:55.003+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:55.003+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:55.502 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:55.409+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:55.502 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:55.409+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:55.502 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:55.409+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:55.792 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:49:55.500+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21904 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:49:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:56.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:56.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:56.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:55.790+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:55.790+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:55.790+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:56.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:56.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:56.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:56.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:56.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:56.449+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:57.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:57.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:57.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:57.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:56.808+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:56.808+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:56.808+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:57.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:57.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:57.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:57.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:57.411+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:58.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:58.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:58.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:57.834+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:57.834+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:57.834+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:58.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:58.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:58.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:58.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:58.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:58.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:59.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:58.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:59.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:58.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:59.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:58.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:49:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:58.842+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:58.842+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:58.842+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:49:59.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:59.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:59.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:59.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:49:59.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:49:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:49:59.399+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:00.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:59.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:59.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:49:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:49:59.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:00.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:59.816+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:00.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:59.816+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:00.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:49:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:49:59.816+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:00.724 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:00.503+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 21975 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:00.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:00.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:00.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:00.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:00.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:00.445+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:00.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:00.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:00.793+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:01.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:01.001+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:01.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:01.001+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:01.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:01.001+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:01.801 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:01.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:01.801 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:01.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:01.802 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:01.492+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:01.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:01.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:02.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:01.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:02.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:01.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:02.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:01.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:02.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:01.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:02.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:02.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:02.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:02.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:02.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:02.505+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:02.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:02.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:02.796+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:02.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:03.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:02.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:03.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:02.957+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:03.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:03.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:03.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:03.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:03.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:03.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:04.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:03.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:04.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:03.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:04.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:03.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:03.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:03.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:04.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:03.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:04.801 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:04.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:04.802 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:04.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:04.802 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:04.519+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:05.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:04.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:05.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:04.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:05.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:04.800+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:04.906+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:04.906+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:04.906+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:05.632 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:05.507+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22047 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:05.632 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:05.537+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:05.632 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:05.537+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:05.633 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:05.537+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:05.946 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:05.820+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:05.946 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:05.820+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:05.946 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:05.820+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:06.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:05.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:05.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:05.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:06.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:06.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:06.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:06.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:06.796 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:06.489+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:06.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:06.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:06.795+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:07.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:06.933+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:07.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:06.933+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:07.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:06.933+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:07.778 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:07.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:07.778 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:07.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:07.779 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:07.516+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:07.977+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:07.977+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:07.977+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:08.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:07.777+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:08.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:07.777+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:08.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:07.777+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:08.939 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:08.728+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:08.940 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:08.728+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:08.940 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:08.728+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:08.940 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:08.538+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:08.941 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:08.538+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:08.941 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:08.538+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:08.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:08.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:09.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:08.938+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:09.947 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:09.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:09.947 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:09.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:09.947 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:09.723+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:09.947 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:09.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:09.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:09.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:09.948 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:09.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:10.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:09.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:10.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:09.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:10.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:09.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:10.767 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:10.510+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22117 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:10.767 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:10.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:10.768 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:10.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:10.768 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:10.523+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:10.902+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:11.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:10.902+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:11.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:10.902+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:11.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:10.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:11.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:10.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:11.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:10.766+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:11.865 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:11.797+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:11.866 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:11.797+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:11.866 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:11.797+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:11.866 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:11.564+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:11.866 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:11.564+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:11.867 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:11.564+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:11.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:11.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:12.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:11.864+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:12.887 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:12.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:12.887 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:12.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:12.887 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:12.771+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:12.888 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:12.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:12.888 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:12.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:12.888 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:12.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:12.885+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:12.885+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:12.885+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:13.853+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:13.853+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:13.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:13.853+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:13.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:13.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:13.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:13.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:13.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:13.745+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:13.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:13.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:13.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:13.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:14.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:14.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:14.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:14.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:14.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:14.715+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:14.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:14.715+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:14.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:14.715+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:14.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:14.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:14.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:14.645+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:15.625 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:15.514+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22189 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:15.916 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:15.719+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:15.916 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:15.719+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:15.917 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:15.719+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:15.917 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:15.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:15.917 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:15.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:15.917 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:15.623+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:16.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:15.915+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:16.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:15.915+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:16.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:15.915+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:16.901 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:16.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:16.901 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:16.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:16.901 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:16.738+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:16.901 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:16.614+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:16.901 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:16.614+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:16.902 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:16.614+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:16.899+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:16.899+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:16.899+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:17.892 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:17.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:17.892 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:17.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:17.892 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:17.629+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:17.893 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:17.731+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:17.893 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:17.731+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:17.893 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:17.731+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:17.891+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:17.891+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:17.891+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:18.905 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:18.596+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:18.905 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:18.596+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:18.906 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:18.596+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:18.906 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:18.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:18.906 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:18.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:18.906 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:18.729+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:18.904+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:18.904+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:18.904+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:19.916 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:19.591+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:19.917 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:19.591+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:19.917 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:19.591+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:19.917 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:19.683+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:19.917 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:19.683+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:19.917 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:19.683+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:19.915+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:19.915+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:19.915+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:20.901 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:20.517+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22260 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:20.902 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:20.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:20.902 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:20.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:20.902 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:20.689+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:20.902 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:20.579+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:20.902 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:20.579+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:20.902 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:20.579+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:21.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:20.900+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:20.900+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:21.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:20.900+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:21.945 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:21.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:21.945 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:21.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:21.945 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:21.561+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:21.945 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:21.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:21.946 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:21.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:21.946 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:21.733+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:22.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:21.944+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:21.944+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:21.944+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:22.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:22.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:22.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:22.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:22.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:22.552+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:22.974 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:22.768+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:22.974 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:22.768+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:22.974 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:22.768+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:22.972+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:22.972+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:23.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:22.972+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:23.882 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:23.789+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:23.883 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:23.789+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:23.883 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:23.789+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:23.883 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:23.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:23.883 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:23.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:23.883 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:23.553+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:23.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:23.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:24.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:23.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:24.821 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:24.566+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:24.821 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:24.566+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:24.821 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:24.566+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:24.819+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:24.819+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:24.819+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:25.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:25.004+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:25.004+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:25.004+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:25.836 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:25.521+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22332 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:25.836 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:25.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:25.837 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:25.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:25.837 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:25.570+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:25.835+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:25.835+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:25.835+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:26.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:26.011+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:26.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:26.011+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:26.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:26.011+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:26.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:26.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:26.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:26.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:26.802+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:26.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:26.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:26.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:26.592+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:27.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:27.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:27.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:27.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:27.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:27.638+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:27.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:27.776+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:27.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:27.776+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:27.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:27.776+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:28.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:28.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:28.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:28.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:28.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:28.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:28.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:28.817+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:28.656+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:28.656+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:28.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:28.656+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:29.054+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:29.054+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:29.054+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:29.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:29.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:29.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:29.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:29.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:29.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:29.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:29.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:29.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:29.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:29.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:29.857+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:30.061+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:30.061+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:30.061+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:30.855 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:30.524+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22402 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:30.856 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:30.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:30.856 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:30.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:30.856 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:30.618+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:30.855+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:30.855+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:30.855+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:31.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:31.054+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:31.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:31.054+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:31.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:31.054+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:31.897 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:31.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:31.897 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:31.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:31.898 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:31.610+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:31.895+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:31.895+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:32.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:31.895+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:32.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:32.056+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:32.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:32.056+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:32.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:32.056+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:32.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:32.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:32.643+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:32.878+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:32.878+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:32.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:32.878+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:33.088+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:33.088+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:33.088+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:33.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:33.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:33.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:33.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:33.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:33.689+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:33.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:33.864+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:33.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:33.864+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:33.976 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:33.864+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:34.120+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:34.120+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:34.120+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:35.080 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:34.844+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:35.080 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:34.844+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:35.081 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:34.844+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:35.081 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:34.729+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:35.081 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:34.729+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:35.081 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:34.729+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:35.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:35.079+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:35.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:35.079+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:35.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:35.079+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:35.881 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:35.527+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22474 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:35.881 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:35.694+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:35.881 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:35.694+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:35.881 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:35.694+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:35.880+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:35.880+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:36.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:35.880+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:36.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:36.043+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:36.043+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:36.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:36.043+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:36.929 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:36.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:36.930 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:36.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:36.930 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:36.673+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:36.929+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:36.929+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:37.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:36.929+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:37.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:37.010+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:37.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:37.010+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:37.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:37.010+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:37.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:37.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:37.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:37.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:37.973 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:37.661+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:38.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:37.972+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:38.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:37.972+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:38.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:37.972+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:38.007+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:38.007+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:38.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:38.007+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:38.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:38.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:38.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:38.648+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:39.018+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:39.018+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:39.018+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:39.020+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:39.020+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:39.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:39.020+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:39.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:39.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:39.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:39.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:39.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:39.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:40.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:40.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:40.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:40.048+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:40.058+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:40.058+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:40.058+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:40.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:40.530+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22545 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:40.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:40.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:40.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:41.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:41.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:41.049+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:41.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:41.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:41.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:41.615+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:41.615+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:41.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:41.615+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:42.314 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:41.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:42.314 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:41.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:42.315 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:41.976+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:42.315 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:42.012+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:42.315 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:42.012+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:42.315 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:42.012+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:42.932 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:42.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:42.933 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:42.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:42.933 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:42.654+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:43.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:42.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:42.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:43.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:42.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:43.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:42.989+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:43.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:42.989+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:43.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:42.989+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:43.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:43.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:43.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:43.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:43.963 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:43.626+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:43.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:43.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:44.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:43.995+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:43.961+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:43.961+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:44.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:43.961+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:44.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:44.632+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:44.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:44.632+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:44.928 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:44.632+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:45.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:44.926+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:45.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:44.926+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:45.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:44.926+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:45.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:45.012+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:45.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:45.012+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:45.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:45.012+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:45.556 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:45.533+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22617 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:45.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:45.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:45.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:45.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:45.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:45.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:46.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:45.971+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:45.971+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:45.971+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:46.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:46.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:46.045+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:46.954 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:46.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:46.954 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:46.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:46.954 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:46.612+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:47.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:46.952+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:46.952+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:46.952+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:47.058+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:47.058+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:47.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:47.058+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:47.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:47.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:47.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:47.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:47.582+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:47.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:47.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:48.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:47.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:48.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:48.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:48.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:48.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:48.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:48.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:48.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:48.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:48.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:48.539+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:49.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:49.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:49.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:48.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:48.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:49.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:48.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:49.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:49.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:49.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:49.573+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:50.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:50.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:50.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:50.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:50.086+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:50.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:50.086+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:50.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:50.086+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:50.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:50.536+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22687 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:50.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:50.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:50.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:50.571+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:51.010+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:51.010+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:51.010+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:51.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:51.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:51.055+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:51.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:51.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:51.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:51.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:51.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:51.530+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:52.035+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:52.035+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:52.035+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:51.979+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:51.979+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:52.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:51.979+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:52.967 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:52.577+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:52.968 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:52.577+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:52.968 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:52.577+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:52.966+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:52.966+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:52.966+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:53.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:53.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:53.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:53.967 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:53.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:53.967 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:53.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:53.967 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:53.532+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:53.966+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:53.966+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:54.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:53.966+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:53.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:53.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:53.995+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:54.970 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:54.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:54.971 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:54.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:54.971 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:54.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:55.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:55.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:55.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:54.969+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:54.969+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:55.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:54.969+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:55.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:50:55.541+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22759 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:50:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:55.538+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:55.538+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:55.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:55.538+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:56.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:55.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:55.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:56.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:55.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:56.076+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:56.076+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:56.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:56.076+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:56.952 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:56.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:56.953 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:56.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:56.953 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:56.511+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:57.056+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:57.056+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:57.056+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:56.951+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:56.951+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:56.951+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:57.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:57.507+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:57.507+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:57.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:57.507+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:58.087+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:58.087+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:58.087+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:57.990+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:57.990+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:58.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:57.990+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:58.499+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:58.499+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:58.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:58.499+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:59.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:59.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:59.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:59.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:50:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:50:59.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:50:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:59.040+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:59.040+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:50:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:50:59.040+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:50:59.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:59.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:59.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:59.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:50:59.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:50:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:50:59.485+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:00.058+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:00.058+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:00.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:00.058+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:00.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:00.067+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:00.067+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:00.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:00.067+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:00.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:00.544+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22830 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:00.482+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:00.482+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:00.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:00.482+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:01.437 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:01.107+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:01.437 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:01.107+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:01.438 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:01.107+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:01.438 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:01.022+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:01.438 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:01.022+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:01.438 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:01.022+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:01.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:01.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:01.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:01.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:01.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:01.436+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:02.465 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:02.105+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:02.466 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:02.105+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:02.466 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:02.105+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:02.466 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:02.061+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:02.466 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:02.061+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:02.466 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:02.061+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:02.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:02.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:02.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:02.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:02.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:02.464+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:03.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:03.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:03.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:03.147+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:03.074+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:03.074+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:03.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:03.074+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:03.818 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:03.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:03.818 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:03.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:03.818 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:03.490+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:04.456 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:04.185+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:04.457 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:04.185+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:04.457 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:04.185+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:04.457 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:04.108+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:04.457 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:04.108+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:04.457 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:04.108+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:04.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:04.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:04.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:04.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:04.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:04.455+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:05.439 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:05.231+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:05.439 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:05.231+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:05.439 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:05.231+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:05.439 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:05.090+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:05.439 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:05.090+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:05.440 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:05.090+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:05.725 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:05.547+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22898 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:05.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:05.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:05.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:05.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:05.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:05.437+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:06.434 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:06.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:06.434 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:06.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:06.434 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:06.220+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:06.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:06.110+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:06.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:06.110+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:06.435 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:06.110+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:06.432+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:06.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:06.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:06.433+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:07.416 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:07.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:07.416 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:07.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:07.416 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:07.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:07.416 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:07.102+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:07.417 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:07.102+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:07.417 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:07.102+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:07.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:07.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:07.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:07.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:07.414+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:08.436 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:08.230+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:08.436 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:08.230+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:08.436 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:08.230+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:08.437 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:08.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:08.437 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:08.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:08.437 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:08.151+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:08.435+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:08.435+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:08.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:08.435+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:09.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:09.160+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:09.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:09.160+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:09.452 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:09.160+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:09.452 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:09.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:09.453 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:09.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:09.453 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:09.256+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:09.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:09.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:09.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:09.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:09.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:09.450+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:10.264+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:10.264+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:10.264+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:10.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:10.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:10.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:10.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:10.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:10.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:10.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:10.551+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 22968 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:10.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:10.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:10.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:10.474+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:11.241+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:11.241+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:11.241+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:11.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:11.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:11.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:11.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:11.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:11.134+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:11.746 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:11.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:11.746 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:11.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:11.746 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:11.494+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:12.468 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:12.206+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:12.468 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:12.206+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:12.468 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:12.206+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:12.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:12.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:12.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:12.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:12.469 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:12.131+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:12.466+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:12.466+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:12.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:12.467+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:13.466 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:13.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:13.466 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:13.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:13.466 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:13.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:13.467 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:13.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:13.467 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:13.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:13.467 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:13.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:13.724 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:13.465+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:13.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:13.465+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:13.725 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:13.465+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:14.238+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:14.238+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:14.238+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:14.128+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:14.128+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:14.128+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:14.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:14.483+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:14.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:14.483+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:14.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:14.483+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:15.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:15.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:15.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:15.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:15.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:15.190+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:15.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:15.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:15.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:15.974 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:15.554+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23040 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:15.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:15.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:15.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:15.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:15.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:15.493+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:16.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:16.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:16.204+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:16.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:16.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:16.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:16.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:16.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:16.543+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:16.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:16.543+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:16.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:16.543+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:17.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:17.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:17.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:17.217+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:17.119+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:17.119+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:17.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:17.119+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:17.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:17.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:17.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:17.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:17.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:17.563+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:18.266+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:18.266+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:18.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:18.266+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:18.097+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:18.097+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:18.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:18.097+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:18.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:18.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:18.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:18.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:18.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:18.603+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:19.286+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:19.286+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:19.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:19.286+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:19.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:19.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:19.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:19.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:19.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:19.089+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:19.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:19.619+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:19.619+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:19.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:19.619+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:20.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:20.277+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:20.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:20.277+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:20.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:20.277+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:20.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:20.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:20.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:20.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:20.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:20.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:20.558+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23111 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:20.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:20.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:20.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:20.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:20.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:20.627+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:21.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:21.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:21.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:21.283+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:21.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:21.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:21.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:21.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:21.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:21.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:21.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:21.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:21.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:21.598+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:22.301+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:22.301+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:22.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:22.301+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:22.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:22.130+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:22.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:22.130+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:22.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:22.130+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:22.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:22.587+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:22.587+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:22.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:22.587+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:23.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:23.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:23.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:23.162+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:23.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:23.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:23.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:23.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:23.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:23.253+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:23.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:23.549+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:23.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:23.549+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:23.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:23.549+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:24.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:24.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:24.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:24.251+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:24.174+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:24.174+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:24.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:24.174+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:24.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:24.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:24.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:24.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:24.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:24.585+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:25.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:25.233+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:25.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:25.233+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:25.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:25.233+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:25.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:25.209+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:25.209+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:25.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:25.209+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:25.974 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:25.561+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23183 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:25.606+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:25.607+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:25.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:25.607+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:26.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:26.252+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:26.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:26.252+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:26.635 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:26.252+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:26.636 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:26.236+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:26.636 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:26.236+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:26.636 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:26.236+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:26.634+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:26.634+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:26.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:26.634+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:27.287+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:27.287+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:27.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:27.287+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:27.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:27.221+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:27.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:27.221+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:27.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:27.221+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:27.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:27.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:27.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:27.608+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:28.188 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-02-16T22:51:28.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-16T22:51:28.193 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-16T22:51:28.193 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-16T22:51:28.205 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-02-16T22:51:28.215 ERROR:teuthology.run_tasks:Manager failed: full_sequential_finally Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/full_sequential_finally.py", line 38, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/full_sequential_finally.py", line 43, in task ((taskname, confg),) = entry.items() ValueError: too many values to unpack (expected 1) 2024-02-16T22:51:28.224 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-16T22:51:28.240 INFO:tasks.cephadm:Teardown begin 2024-02-16T22:51:28.241 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-16T22:51:28.252 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-16T22:51:28.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:51:28.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:51:28.627 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:28.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:28.627 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:28.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:28.628 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:28.242+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:28.628 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:28.234+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:28.628 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:28.234+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:28.628 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:28.234+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:28.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:28.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:28.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:28.625+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:29.215+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:29.215+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:29.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:29.215+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:29.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:29.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:29.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:29.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:29.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:29.202+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:29.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:29.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:29.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:29.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:29.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:29.576+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:30.195+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:30.195+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:30.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:30.195+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:30.172+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:30.172+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:30.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:30.172+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:30.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:30.563+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23253 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:30.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:30.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:30.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:30.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:30.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:30.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:31.356 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:51:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:31.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:31.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:31.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:31.181+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:31.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:31.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:31.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:31.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:31.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:31.167+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:31.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:31.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:31.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:31.621+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:32.168+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:32.168+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:32.168+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:32.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:32.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:32.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:32.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:32.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:32.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:32.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:32.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:32.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:32.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:32.655+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:33.136+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:33.136+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:33.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:33.136+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:33.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:33.185+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:33.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:33.185+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:33.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:33.185+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:33.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:33.693+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:33.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:33.693+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:33.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:33.693+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:34.141+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:34.141+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:34.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:34.141+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:34.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:34.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:34.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:34.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:34.178+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:34.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:34.743+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:34.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:34.743+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:34.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:34.743+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:35.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:35.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:35.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:35.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:35.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:35.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:35.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:35.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:35.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:35.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:35.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:35.154+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:35.974 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:35.567+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23325 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:35.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:35.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:35.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:35.758+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:36.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:36.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:36.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:36.191+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:36.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:36.191+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:36.191+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:37.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:36.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:37.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:36.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:37.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:36.777+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:37.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:37.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:37.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:37.230+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:37.230+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:37.230+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:38.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:37.824+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:38.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:37.824+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:38.085 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:37.824+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:38.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:38.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:38.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:38.247+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:38.247+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:38.247+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:39.068 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:38.796+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:39.068 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:38.796+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:39.068 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:38.796+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:39.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:39.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:39.067+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:39.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:39.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:39.266+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:40.067 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:39.770+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:40.068 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:39.771+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:40.068 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:39.771+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:40.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:51:40.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:51:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:40.066+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:40.066+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:40.066+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:40.237+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:40.237+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:40.237+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:40.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:40.571+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23396 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:40.749+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:40.749+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:40.749+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:41.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:41.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:41.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:41.080+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:41.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:41.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:41.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:41.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:41.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:41.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:42.039 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:41.739+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:42.040 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:41.739+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:42.040 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:41.739+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:42.038+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:42.038+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:42.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:42.038+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:42.235+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:42.235+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:42.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:42.235+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:43.191 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:43.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:43.192 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:43.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:43.192 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:43.005+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:43.192 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:42.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:43.192 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:42.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:43.193 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:42.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:43.420 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:51:43.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:43.190+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:43.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:43.190+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:43.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:43.190+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:44.020 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:43.778+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:44.020 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:43.778+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:44.020 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:43.778+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:44.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:44.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:44.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:44.018+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:44.018+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:44.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:44.018+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:45.147 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:44.984+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:45.147 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:44.984+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:45.147 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:44.984+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:45.148 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:44.799+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:45.148 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:44.799+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:45.148 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:44.799+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:45.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:45.146+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:45.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:45.146+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:45.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:45.146+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:45.842 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:45.575+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23468 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:46.183 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:45.962+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:46.183 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:45.962+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:46.184 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:45.962+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:46.184 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:45.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:46.184 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:45.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:46.184 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:45.841+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:46.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:46.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:46.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:46.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:47.206 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:47.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:47.207 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:47.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:47.207 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:47.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:47.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:46.815+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:47.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:46.815+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:47.208 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:46.815+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:47.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:47.205+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:47.205+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:47.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:47.205+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:48.190 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:48.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:48.190 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:48.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:48.190 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:48.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:48.190 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:47.853+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:48.191 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:47.853+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:48.191 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:47.853+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:48.189+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:48.189+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:48.189+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:49.171 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:49.033+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:49.171 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:49.033+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:49.171 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:49.033+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:49.171 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:48.880+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:49.171 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:48.880+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:49.172 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:48.880+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:49.169+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:49.169+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:49.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:49.169+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:50.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:50.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:50.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:50.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:50.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:50.125+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:50.125+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:50.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:50.125+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:50.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:49.912+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:50.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:49.912+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:50.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:49.912+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:50.935 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:50.578+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23538 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:51.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:50.934+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:51.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:50.934+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:51.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:50.934+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:50.985+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:50.985+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:50.985+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:51.102+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:51.102+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:51.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:51.102+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:52.000 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:51.933+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:52.000 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:51.933+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:52.000 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:51.933+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:51.999+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:51.999+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:51.999+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:52.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:52.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:52.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:52.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:52.137+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:53.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:52.944+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:53.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:52.944+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:53.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:52.944+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:52.978+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:52.978+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:53.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:52.978+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:53.152+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:53.152+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:53.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:53.152+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:53.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:53.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:53.989+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:54.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:53.916+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:54.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:53.916+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:54.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:53.916+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:54.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:54.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:54.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:55.074+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:55.074+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:55.074+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:55.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:54.927+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:55.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:54.927+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:55.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:54.927+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:55.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:55.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:55.016+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:55.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:51:55.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:51:55.967 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:51:55.580+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23610 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:51:56.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:55.966+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:56.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:55.966+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:56.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:55.966+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:55.975+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:55.975+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:55.975+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:56.057+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:56.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:56.057+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:56.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:56.057+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:56.983+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:56.983+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:56.983+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:57.068+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:57.068+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:57.068+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:56.945+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:56.945+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:56.945+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:58.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:58.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:58.012+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:58.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:58.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:58.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:58.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:57.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:58.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:57.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:58.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:57.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:58.492 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:51:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:59.047+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:59.047+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:51:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:51:59.047+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:51:59.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:59.007+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:59.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:59.007+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:59.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:51:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:51:59.007+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:51:59.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:58.879+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:59.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:58.879+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:51:59.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:58.879+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:00.024+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:00.024+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:00.024+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:00.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:59.925+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:00.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:59.925+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:00.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:51:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:51:59.925+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:00.004+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:00.004+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:00.004+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:00.886 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:00.584+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23681 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:01.027+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:01.027+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:01.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:01.027+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:01.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:00.885+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:01.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:00.885+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:01.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:00.885+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:01.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:01.049+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:01.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:01.049+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:01.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:01.049+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:01.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:52:01.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:52:01.564 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:52:01.564 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:52:01.564 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-16T22:52:01.565 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-16T22:52:01.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:52:01.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:52:02.009 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:01.925+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:02.009 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:01.925+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:02.009 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:01.925+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:02.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:02.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:02.008+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:02.065+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:02.065+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:02.065+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:02.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:02.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:03.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:02.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:03.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:03.056+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:03.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:03.056+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:03.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:03.056+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:03.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:02.885+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:03.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:02.885+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:03.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:02.885+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:03.934+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:03.934+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:04.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:03.934+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:04.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:04.028+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:04.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:04.028+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:04.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:04.028+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:04.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:03.930+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:04.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:03.930+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:04.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:03.930+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:04.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:52:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:04.951+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:04.951+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:05.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:04.951+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:05.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:04.925+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:05.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:04.925+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:05.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:04.925+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:05.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:05.071+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:05.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:05.071+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:05.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:05.071+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:05.852 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:05.587+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23753 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:06.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:05.887+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:06.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:05.887+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:06.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:05.887+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:05.985+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:06.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:05.985+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:06.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:05.985+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:06.035+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:06.035+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:06.035+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:07.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:07.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:07.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:07.050+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:07.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:06.879+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:07.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:06.879+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:07.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:06.879+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:07.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:06.961+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:07.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:06.961+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:07.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:06.961+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:07.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:07.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:08.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:07.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:08.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:08.098+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:08.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:08.098+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:08.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:08.098+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:08.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:07.908+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:08.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:07.908+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:08.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:07.908+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:09.129 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:08.958+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:09.129 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:08.958+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:09.129 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:08.958+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:09.130 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:08.868+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:09.130 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:08.868+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:09.130 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:08.868+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:09.426 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:09.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:09.426 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:09.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:09.426 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:09.127+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:10.034 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:09.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:10.035 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:09.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:10.035 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:09.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:10.035 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:09.909+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:10.035 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:09.909+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:10.035 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:09.909+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:10.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:10.175+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:10.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:10.175+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:10.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:10.175+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:10.925 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:10.591+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23823 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:11.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:11.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:11.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:11.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:11.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:11.173+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:11.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:10.924+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:11.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:10.924+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:11.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:10.924+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:11.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:10.945+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:11.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:10.945+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:11.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:10.945+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:12.163 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:11.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:12.163 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:11.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:12.163 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:11.930+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:12.163 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:11.922+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:12.163 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:11.922+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:12.164 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:11.922+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:12.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:12.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:12.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:12.161+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:13.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:13.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:13.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:13.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:13.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:13.121+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:12.888+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:13.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:12.888+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:13.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:12.888+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:13.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:12.878+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:13.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:12.878+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:13.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:12.878+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:13.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:52:13.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:52:14.114 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:13.881+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:14.114 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:13.881+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:14.114 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:13.881+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:14.114 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:13.852+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:14.115 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:13.852+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:14.115 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:13.852+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:14.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:14.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:14.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:14.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:14.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:14.112+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:15.144 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:14.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:15.144 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:14.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:15.144 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:14.846+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:15.144 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:14.852+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:15.145 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:14.852+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:15.145 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:14.852+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:15.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:15.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:15.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:15.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:15.894 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:15.593+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23895 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:15.895 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:15.842+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:15.895 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:15.842+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:15.895 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:15.842+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:16.093+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:16.093+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:16.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:16.093+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:16.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:15.893+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:16.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:15.893+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:16.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:15.893+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:16.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:52:17.120 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:16.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:17.120 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:16.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:17.120 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:16.865+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:17.120 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:16.870+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:17.120 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:16.870+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:17.121 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:16.870+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:17.118+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:17.118+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:17.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:17.118+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:17.896+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:17.896+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:17.896+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:18.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:18.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:18.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:18.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:18.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:18.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:18.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:17.900+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:18.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:17.900+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:18.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:17.900+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:18.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:18.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:18.850+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:19.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:18.934+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:19.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:18.934+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:19.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:18.934+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:19.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:19.081+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:19.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:19.081+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:19.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:19.081+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:19.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:19.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:19.807+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:20.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:20.033+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:20.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:20.033+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:20.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:20.033+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:20.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:19.929+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:20.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:19.929+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:20.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:19.929+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:20.926 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:20.596+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 23966 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:20.926 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:20.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:20.926 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:20.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:20.927 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:20.810+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:21.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:21.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:21.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:21.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:21.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:20.924+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:21.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:20.924+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:21.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:20.924+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:21.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:21.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:22.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:21.822+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:21.982+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:22.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:21.982+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:22.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:21.982+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:22.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:21.949+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:22.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:21.949+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:22.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:21 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:21.949+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:23.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:22.939+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:23.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:22.939+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:23.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:22.939+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:23.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:22.954+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:23.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:22.954+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:23.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:22.954+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:22.832+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:22.832+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:23.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:22 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:22.832+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:23.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:23.914+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:23.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:23.914+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:23.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:23.914+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:23.975 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:23.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:23.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:23.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:23.976 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:23.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:23.938+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:23.938+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:23.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:23 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:23.938+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:24.878+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:24.878+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:25.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:24.878+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:24.907+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:24.907+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:25.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:24.907+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:25.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:24.961+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:25.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:24.961+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:25.227 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:24 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:24.961+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:25.877 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:25.600+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24038 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:25.900+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:25.900+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:26.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:25.900+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:26.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:25.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:26.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:25.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:26.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:25.875+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:26.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:25.965+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:26.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:25.965+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:26.227 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:25.965+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:27.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:26.940+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:27.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:26.940+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:27.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:26.940+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:27.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:26.860+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:27.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:26.860+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:27.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:26.860+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:27.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:26.947+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:27.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:26.947+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:27.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:26 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:26.947+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:27.986+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:27.986+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:28.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:27.986+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:27.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:28.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:27.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:28.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:27.871+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:28.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:27.941+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:28.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:27.941+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:28.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:27 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:27.941+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:28.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:52:28.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:52:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:29.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:29.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:29.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:29.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:29.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:28.827+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:29.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:28.827+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:29.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:28.827+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:29.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:28.962+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:29.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:28.962+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:29.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:28 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:28.962+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:29.874+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:29.874+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:30.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:29.874+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:30.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:29.931+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:30.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:29.931+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:30.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:29 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:29.931+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:30.074+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:30.074+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:30.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:30.074+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:30.893 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:30.603+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24108 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:30.892+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:30.892+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:31.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:30.892+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:31.060+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:31.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:31.060+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:31.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:31.060+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:31.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:30.968+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:31.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:30.968+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:31.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:30.968+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:31.776 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:52:32.006 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:31.935+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:32.006 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:31.935+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:32.006 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:31 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:31.935+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:32.043+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:32.043+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:32.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:32.043+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:32.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:32.005+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:32.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:32.005+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:32.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:32.005+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:33.057+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:33.057+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:33.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:33.057+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:32.970+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:33.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:32.970+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:33.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:32 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:32.970+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:33.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:33.023+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:33.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:33.023+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:33.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:33.024+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:34.021+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:34.021+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:34.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:34.021+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:33.966+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:33.966+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:34.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:33 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:33.966+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:34.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:34.037+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:34.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:34.037+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:34.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:34.037+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:34.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:52:34.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:52:34.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:52:34.844 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:52:34.844 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-16T22:52:34.846 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728/remote/smithi002/crash 2024-02-16T22:52:34.847 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/crash -- . 2024-02-16T22:52:34.926 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728/remote/smithi103/crash 2024-02-16T22:52:34.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:52:34.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:52:35.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:34.983+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:35.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:34.983+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:35.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:34.983+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:35.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:34.982+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:35.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:34.982+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:35.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:34 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:34.982+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:35.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:35.021+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:35.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:35.021+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:35.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:35.021+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:35.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:35.607+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24180 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:35.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:35.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:36.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:35.997+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:35.974+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:35.974+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:36.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:35.974+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:36.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:36.040+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:36.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:36.040+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:36.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:36 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:36.040+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:37.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:37.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:37.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:37.017+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:37.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:37.022+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:37.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:37.023+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:37.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:37.023+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:37.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:37.061+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:37.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:37.061+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:37.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:37.061+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:37.980 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:52:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:38.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:38.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:38.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:38.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:38.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:38.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:38.017+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:38.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:38.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:38.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:38.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:38.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:38.088+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:39.035+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:39.035+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:39.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:39.035+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:39.020+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:39.020+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:39.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:39.020+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:39.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:39.067+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:39.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:39.067+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:39.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:39.067+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:40.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:40.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:40.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:40.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:40.069+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:40.069+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:40.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:40.069+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:40.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:40.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:40.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:40.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:40.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:40.042+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:40.974 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:40.610+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24251 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:41.115+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:41.115+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:41.115+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:41.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:41.002+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:41.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:41.002+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:41.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:41.002+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:41.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:41.065+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:41.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:41.065+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:41.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:41.065+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:42.071+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:42.071+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:42.071+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:42.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:42.015+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:42.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:42.015+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:42.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:42.015+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:42.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:42.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:42.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:42.027+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:42.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:42.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:43.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:42.980+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:43.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:43.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:43.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:43.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:43.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:43.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:43.016+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:43.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:43.016+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:43.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:43.016+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:44.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:44.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:44.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:44.002+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:44.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:44.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:44.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:44.025+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:44.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:43.996+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:44.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:43.996+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:44.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:43.996+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:45.302 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:44.990+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:45.302 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:44.990+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:45.302 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:44.990+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:44.992+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:44.992+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:45.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:44.992+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:45.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:45.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:45.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:45.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:45.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:45.037+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:45.946 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:45.614+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24323 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:45.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:45.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:46.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:45.945+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:46.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:46.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:46.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:46.046+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:46.002+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:46.002+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:46.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:46.002+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:46.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:52:46.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:52:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:46.947+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:46.947+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:47.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:46.947+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:47.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:46.973+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:47.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:46.973+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:47.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:46.973+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:47.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:47.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:47.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:47.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:47.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:47.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:47.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:47.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:48.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:47.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:47.959+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:48.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:47.959+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:48.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:47.959+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:48.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:48.004+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:48.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:48.004+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:48.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:48.004+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:48.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:48.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:49.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:48.968+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:48.999+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:48.999+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:49.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:48.999+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:49.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:49.000+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:49.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:49.000+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:49.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:49.000+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:50.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:52:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:49.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:49.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:50.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:49.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:50.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:50.030+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:50.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:50.030+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:50.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:50.030+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:50.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:49.983+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:50.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:49.983+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:50.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:49.983+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:50.970 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:50.617+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24393 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:51.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:50.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:50.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:51.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:50.994+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:51.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:51.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:51.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:51.009+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:51.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:50.968+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:51.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:50.968+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:51.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:50.968+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:52.008 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:51.937+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:52.008 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:51.937+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:52.008 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:51.937+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:52.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:52.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:52.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:52.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:52.022+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:52.007+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:52.007+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:52.007+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:53.224 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:53.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:53.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:53.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:53.001+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:53.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:52.940+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:53.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:52.940+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:53.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:52.940+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:53.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:53.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:53.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:53.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:53.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:53.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:54.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:54.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:54.009+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:54.010+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:54.010+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:54.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:54.010+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:54.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:53.938+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:54.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:53.938+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:54.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:53.938+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:54.984+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:54.984+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:55.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:54.984+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:55.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:54.907+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:55.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:54.907+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:55.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:54.907+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:55.030+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:55.030+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:55.030+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:55.944 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:52:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:52:55.621+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24465 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:52:56.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:56.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:56.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:56.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:56.072+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:56.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:56.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:56.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:56.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:56.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:55.943+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:56.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:55.943+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:56.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:55.943+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:57.106+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:57.106+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:57.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:57.106+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:57.028+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:57.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:57.028+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:57.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:57.028+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:56.914+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:56.914+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:57.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:56.914+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:58.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:58.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:58.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:58.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:58.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:58.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:58.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:58.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:58.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:58.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:57.907+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:58.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:57.907+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:58.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:57.907+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:59.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:59.071+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:59.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:59.071+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:59.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:52:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:52:59.071+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:52:59.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:59.076+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:59.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:59.076+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:59.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:52:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:52:59.076+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:52:59.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:58.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:59.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:58.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:52:59.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:58.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:00.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:00.064+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:00.064+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:00.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:00.064+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:00.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:00.098+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:00.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:00.098+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:00.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:00.098+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:00.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:59.923+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:00.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:59.923+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:00.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:52:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:52:59.923+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:00.921 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:00.624+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24536 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:53:01.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:00.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:01.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:00.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:01.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:00.920+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:01.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:01.069+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:01.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:01.069+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:01.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:01.069+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:01.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:01.084+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:01.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:01.084+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:01.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:01.084+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:02.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:53:02.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:53:02.047 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:01.953+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:02.048 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:01.953+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:02.048 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:01.953+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:02.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:02.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:02.044+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:02.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:02.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:02.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:02.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:02.093+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:03.059+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:03.059+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:03.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:03.059+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:03.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:02.956+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:03.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:02.956+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:03.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:02.956+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:03.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:03.065+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:03.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:03.065+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:03.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:03.065+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:04.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:04.076+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:04.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:04.076+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:04.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:04.076+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:04.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:03.940+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:04.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:03.940+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:04.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:03.940+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:04.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:04.052+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:04.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:04.052+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:04.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:04.052+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:05.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:53:05.302 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:04.988+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:05.302 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:04.988+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:05.302 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:04.988+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:05.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:05.087+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:05.302 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:05.087+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:05.303 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:05.087+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:05.010+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:05.010+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:05.303 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:05.010+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:05.969 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:05.628+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24608 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:53:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:06.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:06.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:06.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:06.048+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:06.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:06.042+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:06.042+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:06.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:06.042+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:06.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:05.967+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:06.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:05.967+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:06.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:05.967+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:07.071+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:07.071+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:07.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:07.071+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:07.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:07.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:07.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:07.003+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:07.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:06.986+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:07.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:06.986+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:07.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:06.986+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:08.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:53:08.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:53:08.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:53:08.188 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:53:08.188 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-16T22:53:08.189 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-16T22:53:08.219 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:07.950+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:08.220 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:07.950+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:08.220 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:07.950+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:08.220 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:08.061+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:08.220 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:08.061+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:08.221 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:08.061+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:08.221 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:07.983+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:08.221 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:07.983+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:08.221 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:07.983+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:08.224 INFO:teuthology.orchestra.run.smithi002.stdout:1708122000.0001442 mon.smithi002 (mon.0) 772 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-16T22:53:08.225 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-16T22:53:08.225 INFO:tasks.cephadm:Compressing logs... 2024-02-16T22:53:08.226 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-16T22:53:08.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:53:08.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:53:08.295 INFO:teuthology.orchestra.run.smithi002.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-02-16T22:53:08.295 INFO:teuthology.orchestra.run.smithi002.stderr: /var/log/ceph/cephadm.log 2024-02-16T22:53:08.296 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mon.smithi002.log 2024-02-16T22:53:08.296 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mds.cephfs.smithi002.hhufhl.log 2024-02-16T22:53:08.297 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cephadm.log: gzip -5/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mon.smithi002.log: --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.audit.log 2024-02-16T22:53:08.297 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mds.cephfs.smithi002.hhufhl.log: gzip -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.1.log 2024-02-16T22:53:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.audit.log: -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mgr.smithi002.resmjv.log 2024-02-16T22:53:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mds.cephfs.smithi002.llkqse.log 2024-02-16T22:53:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mgr.smithi002.resmjv.log: gzip -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.0.log 2024-02-16T22:53:08.299 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mds.cephfs.smithi002.llkqse.log: gzip -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.cephadm.log 2024-02-16T22:53:08.299 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.0.log: 89.9% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.audit.log.gz 2024-02-16T22:53:08.299 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.log 2024-02-16T22:53:08.300 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.cephadm.log: 81.4% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.cephadm.log.gz 2024-02-16T22:53:08.301 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.2.log 2024-02-16T22:53:08.302 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.log: 86.1% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph.log.gz 2024-02-16T22:53:08.303 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-volume.log 2024-02-16T22:53:08.305 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.2.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-16T22:53:08.317 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-volume.log.gz 2024-02-16T22:53:08.335 INFO:teuthology.orchestra.run.smithi002.stderr: 94.9% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mds.cephfs.smithi002.llkqse.log.gz 2024-02-16T22:53:09.094 INFO:teuthology.orchestra.run.smithi002.stderr:gzip: /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mgr.smithi002.resmjv.log: file size changed while zipping 2024-02-16T22:53:09.095 INFO:teuthology.orchestra.run.smithi002.stderr: 89.9% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mgr.smithi002.resmjv.log.gz 2024-02-16T22:53:09.179 INFO:teuthology.orchestra.run.smithi002.stderr:gzip: /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mon.smithi002.log: file size changed while zipping 2024-02-16T22:53:09.179 INFO:teuthology.orchestra.run.smithi002.stderr: 92.3% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mon.smithi002.log.gz 2024-02-16T22:53:09.270 INFO:teuthology.orchestra.run.smithi002.stderr:gzip: /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-02-16T22:53:09.270 INFO:teuthology.orchestra.run.smithi002.stderr: 93.3% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.2.log.gz 2024-02-16T22:53:09.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:09.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:09.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:09.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:09.045+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:09.006+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:09.006+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:09.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:09.006+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:09.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:08.993+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:09.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:08.993+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:09.476 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:08.993+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:09.655 INFO:teuthology.orchestra.run.smithi002.stderr: 95.4% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-mds.cephfs.smithi002.hhufhl.log.gz 2024-02-16T22:53:09.697 INFO:teuthology.orchestra.run.smithi002.stderr:gzip: /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-02-16T22:53:09.698 INFO:teuthology.orchestra.run.smithi002.stderr: 93.1% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.0.log.gz 2024-02-16T22:53:09.711 INFO:teuthology.orchestra.run.smithi002.stderr:gzip: /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-02-16T22:53:09.711 INFO:teuthology.orchestra.run.smithi002.stderr: 93.3% -- replaced with /var/log/ceph/43c11c36-cd18-11ee-95ba-87774f69a715/ceph-osd.1.log.gz 2024-02-16T22:53:09.713 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-02-16T22:53:09.713 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m1.431s 2024-02-16T22:53:09.713 INFO:teuthology.orchestra.run.smithi002.stderr:user 0m6.736s 2024-02-16T22:53:09.713 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m0.159s 2024-02-16T22:53:10.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:10.015+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:10.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:10.015+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:10.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:10.015+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:10.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:10.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:10.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:10.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:10.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:10.036+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:09.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:09.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:10.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:09.992+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:10.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:10.633+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24678 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:53:11.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:53:11.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:11.045+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:11.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:11.045+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:11.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:11.045+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:11.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:11.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:11.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:11.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:11.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:11.083+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:11.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:10.980+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:11.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:10.980+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:11.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:10.980+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:12.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:12.073+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:12.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:12.073+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:12.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:12.073+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:12.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:12.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:12.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:12.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:12.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:12.081+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:12.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:12.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:12.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:12 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:12.024+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:13.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:13.071+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:13.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:13.071+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:13.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:13.071+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:13.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:13.037+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:13.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:13.037+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:13.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:13.037+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:13.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:13.034+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:13.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:13.034+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:13.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:13.034+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:14.474 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:14.117+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:14.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:14.117+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:14.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:14.117+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:13.988+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:13.988+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:14.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:13 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:13.988+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:14.028+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:14.028+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:14.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:14.028+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:15.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:15.098+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:15.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:15.098+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:15.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:15.098+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:15.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:14.954+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:14.954+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:15.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:14 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:14.954+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:15.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:15.060+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:15.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:15.061+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:15.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:15.061+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:15.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:15.636+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24749 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:53:16.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:16.097+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:16.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:16.097+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:16.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:16.097+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.155728+0000 (oldest deadline 2024-02-16T22:43:05.155728+0000) 2024-02-16T22:53:16.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:15.996+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:16.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:15.996+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:16.476 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:15 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:15.996+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:16.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:16.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:16.476 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:16.036+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.310825+0000 (oldest deadline 2024-02-16T22:43:05.310825+0000) 2024-02-16T22:53:17.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:16.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:16.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:17.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:16 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:16.973+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:18.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:17.931+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:17.931+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:18.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:17 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:17.931+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:18.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:18.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:19.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:18 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:18.941+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:19.926+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:19.926+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:20.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:19 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:19.926+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:42:45.314660+0000 (oldest deadline 2024-02-16T22:43:05.314660+0000) 2024-02-16T22:53:20.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:53:20.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:53:20.974 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:20 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:20.640+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24821 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:53:23.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:53:25.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:25 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:25.643+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24892 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:53:30.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:30 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:30.647+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 24963 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:53:35.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:53:35.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:53:35.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:35 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:35.651+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 25034 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:53:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:37.063+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:37.064+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:37.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:37 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:37.064+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:38.460 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:53:38.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:38.098+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:38.098+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:38.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:38 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:38.098+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:39.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:39.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:39.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:39.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:39 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:39.141+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:40.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:40.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:40.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:40.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:40.094+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:40.975 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:40.654+0000 7ff18a192700 -1 mon.smithi002@0(probing) e2 get_health_metrics reporting 25106 slow ops, oldest is monmgrreport(gid 14212, 0 checks, 1 progress events) 2024-02-16T22:53:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:40.832+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:40.832+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:40.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:40 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:40.832+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:41.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:53:41.461 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:53:41.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:41.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:41.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:41.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:41.122+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:41.532 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:53:41.532 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:53:41.532 INFO:tasks.cephadm:Removing cluster... 2024-02-16T22:53:41.533 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 --force 2024-02-16T22:53:41.822 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:41.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:41.822 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:41.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:41.822 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:41.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:42.122 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:41 smithi002 systemd[1]: Stopping Ceph mon.smithi002 for 43c11c36-cd18-11ee-95ba-87774f69a715... 2024-02-16T22:53:42.122 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:42.069+0000 7ff18d198700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi002 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-16T22:53:42.122 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002[29150]: 2024-02-16T22:53:42.069+0000 7ff18d198700 -1 mon.smithi002@0(probing) e2 *** Got Signal Terminated *** 2024-02-16T22:53:42.122 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:41.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:42.122 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:41.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:42.123 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:41 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:41.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:42.474 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:42 smithi002 podman[84542]: 2024-02-16 22:53:42.283093333 +0000 UTC m=+0.319579344 container died cc6b0b4bb5879d93ff1037254fc5bbb2b726169e6669a5bc06f328c3d072a727 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-b88fae6, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-16T22:53:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:42.120+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:42.120+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:42.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:42.120+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:42.846 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:42.801+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:42.846 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:42.801+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:42.846 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:42.801+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:42.847 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:42 smithi002 podman[84542]: 2024-02-16 22:53:42.686289239 +0000 UTC m=+0.722775267 container cleanup cc6b0b4bb5879d93ff1037254fc5bbb2b726169e6669a5bc06f328c3d072a727 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=quincy-b88fae6, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-16T22:53:42.847 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:42 smithi002 bash[84542]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002 2024-02-16T22:53:43.158 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:42 smithi002 podman[84557]: 2024-02-16 22:53:42.846082465 +0000 UTC m=+0.560987161 container remove cc6b0b4bb5879d93ff1037254fc5bbb2b726169e6669a5bc06f328c3d072a727 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-mon-smithi002, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-b88fae6, org.label-schema.build-date=20240214) 2024-02-16T22:53:43.158 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:43 smithi002 systemd[1]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mon.smithi002.service: Deactivated successfully. 2024-02-16T22:53:43.158 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:43 smithi002 systemd[1]: Stopped Ceph mon.smithi002 for 43c11c36-cd18-11ee-95ba-87774f69a715. 2024-02-16T22:53:43.159 INFO:journalctl@ceph.mon.smithi002.smithi002.stdout:Feb 16 22:53:43 smithi002 systemd[1]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715@mon.smithi002.service: Consumed 22.925s CPU time. 2024-02-16T22:53:43.159 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:42.878+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:43.159 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:42.878+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:43.159 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:42 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:42.878+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:43.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:43.156+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:43.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:43.156+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:43.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:43.156+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:44.156 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:43.927+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:44.157 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:43.927+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:44.157 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:43.927+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:44.157 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:43.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:44.157 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:43.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:44.157 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:43 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:43.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:44.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:44.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:44.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:44.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:44.155+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:45.089 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:44.893+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:45.089 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:44.893+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:45.089 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:44.893+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:45.090 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:44.840+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:45.090 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:44.840+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:45.090 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:44 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:44.840+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:45.414 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:45.165+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:45.414 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:45.165+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:45.414 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:45.165+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:46.151 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:45.850+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:46.152 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:45.850+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:46.152 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:45.850+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:46.152 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:45.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:46.152 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:45.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:46.152 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:45 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:45.871+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:46.149+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:46.149+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:46.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:46.149+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:47.179 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:46.887+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:47.179 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:46.887+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:47.180 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:46.887+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:47.180 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:46.872+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:47.180 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:46.872+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:47.180 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:46 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:46.872+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:47.472 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:47.177+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:47.472 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:47.177+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:47.472 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:47.177+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:48.160 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:47.857+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:48.160 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:47.857+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:48.160 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:47.857+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:48.160 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:47.871+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:48.160 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:47.871+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:48.160 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:47 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:47.871+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:48.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:48.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:48.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:48.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:48.197+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:48.960 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:48.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:48.960 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:48.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:48.960 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:48.849+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:48.960 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:48.839+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:48.960 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:48.839+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:48.961 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:48 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:48.839+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:49.224 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:49.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:49.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:49.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:49.182+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:50.133 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:49.872+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:50.133 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:49.872+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:50.133 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:49.872+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:50.133 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:49.801+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:50.133 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:49.801+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:50.134 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:49 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:49.801+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:50.424 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:50.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:50.424 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:50.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:50.425 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:50.218+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:51.069 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:50.774+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:51.069 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:50.774+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:51.070 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:50.774+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:51.070 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:50.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:51.070 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:50.824+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:51.070 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:50 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:50.825+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:51.070 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:50 smithi002 systemd[1]: Stopping Ceph osd.0 for 43c11c36-cd18-11ee-95ba-87774f69a715... 2024-02-16T22:53:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:51.067+0000 7f2dab1db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-16T22:53:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:51.067+0000 7f2dab1db700 -1 osd.0 46 *** Got signal Terminated *** 2024-02-16T22:53:51.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:51.067+0000 7f2dab1db700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-16T22:53:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:51.210+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:51.210+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:51.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:51.210+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:52.208 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:51.809+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:52.208 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:51.809+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:52.208 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:51.809+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:52.209 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:51.757+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:52.209 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:51.757+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:52.209 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:51 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:51.757+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:52.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:52.206+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:52.206+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:52.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:52.206+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:52.828+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:52.828+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:53.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:52.828+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:53.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:52.760+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:53.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:52.760+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:53.226 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:52 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:52.760+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:53.724 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:53.248+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:53.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:53.248+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:53.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:53.248+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:54.224 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:53.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:53.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:54.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:53.877+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:54.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:53.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:54.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:53.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:54.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:53 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:53.802+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:54.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:54.252+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:54.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:54.252+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:54.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:54.252+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:55.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:54.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:55.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:54.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:55.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:54.773+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:55.225 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:54.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:54.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:55.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:54 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:54.859+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:55.724 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:55.240+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:55.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:55.240+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:55.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:55.240+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:56.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:55.795+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:56.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:55.795+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:56.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:55.795+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:56.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:55.817+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:56.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:55.817+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:56.226 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:55 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0[47727]: 2024-02-16T22:53:55.817+0000 7f2da87db700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.998069+0000 (oldest deadline 2024-02-16T22:53:40.998069+0000) 2024-02-16T22:53:56.514 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:56.244+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:56.514 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:56.244+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:56.514 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:56.244+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:56.796 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:56 smithi002 podman[85395]: 2024-02-16 22:53:56.578994249 +0000 UTC m=+5.598670712 container died a3f19d068bafd21c39aeb0ec7b75c6deb2da91b79bca9822ef596bb73c3e99e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-b88fae6, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-16T22:53:57.214 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:56 smithi002 podman[85395]: 2024-02-16 22:53:56.952234447 +0000 UTC m=+5.971910908 container cleanup a3f19d068bafd21c39aeb0ec7b75c6deb2da91b79bca9822ef596bb73c3e99e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, RELEASE=quincy-b88fae6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-16T22:53:57.214 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:56 smithi002 bash[85395]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0 2024-02-16T22:53:57.215 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:56.795+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:57.215 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:56.795+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:57.215 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:56 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:56.795+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:57.474 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:57 smithi002 podman[85413]: 2024-02-16 22:53:57.214049623 +0000 UTC m=+0.632160645 container remove a3f19d068bafd21c39aeb0ec7b75c6deb2da91b79bca9822ef596bb73c3e99e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-b88fae6, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-02-16T22:53:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:57.239+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:57.239+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:57.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:57.239+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:57.779 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:57 smithi002 podman[85520]: 2024-02-16 22:53:57.420153563 +0000 UTC m=+0.021311239 image pull 870a0944ccb83fae8829e20e516a8aeec00e19c6a392cc9e32528216b9cb93f0 quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c 2024-02-16T22:53:57.780 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:57 smithi002 podman[85520]: 2024-02-16 22:53:57.523273491 +0000 UTC m=+0.124431159 container create 840221c071b17cf392b2a89b2e3ec78108ddf379c45da8d6a44066b5c4b7a794 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0-deactivate, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-b88fae6, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-16T22:53:58.076 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:57.779+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:58.076 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:57.779+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:58.076 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:57 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:57.779+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:58.076 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:57 smithi002 podman[85520]: 2024-02-16 22:53:57.920646039 +0000 UTC m=+0.521803718 container init 840221c071b17cf392b2a89b2e3ec78108ddf379c45da8d6a44066b5c4b7a794 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-b88fae6, org.label-schema.vendor=CentOS) 2024-02-16T22:53:58.077 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:57 smithi002 podman[85520]: 2024-02-16 22:53:57.924198493 +0000 UTC m=+0.525356162 container start 840221c071b17cf392b2a89b2e3ec78108ddf379c45da8d6a44066b5c4b7a794 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240214, RELEASE=quincy-b88fae6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-16T22:53:58.077 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:58 smithi002 podman[85520]: 2024-02-16 22:53:58.075862664 +0000 UTC m=+0.677020341 container attach 840221c071b17cf392b2a89b2e3ec78108ddf379c45da8d6a44066b5c4b7a794 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, RELEASE=quincy-b88fae6, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-16T22:53:58.475 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:58 smithi002 podman[85520]: 2024-02-16 22:53:58.211044363 +0000 UTC m=+0.812202028 container died 840221c071b17cf392b2a89b2e3ec78108ddf379c45da8d6a44066b5c4b7a794 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=quincy-b88fae6, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-16T22:53:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:58.245+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:58.245+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:58.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:58.245+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:59.200 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:58 smithi002 systemd[1]: Stopping Ceph osd.1 for 43c11c36-cd18-11ee-95ba-87774f69a715... 2024-02-16T22:53:59.201 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:58 smithi002 podman[85547]: 2024-02-16 22:53:58.746151209 +0000 UTC m=+0.529821756 container remove 840221c071b17cf392b2a89b2e3ec78108ddf379c45da8d6a44066b5c4b7a794 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-b88fae6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240214) 2024-02-16T22:53:59.201 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:58 smithi002 systemd[1]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-16T22:53:59.201 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:58 smithi002 systemd[1]: Stopped Ceph osd.0 for 43c11c36-cd18-11ee-95ba-87774f69a715. 2024-02-16T22:53:59.201 INFO:journalctl@ceph.osd.0.smithi002.stdout:Feb 16 22:53:58 smithi002 systemd[1]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.0.service: Consumed 17.079s CPU time. 2024-02-16T22:53:59.201 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:58.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:59.202 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:58.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:59.202 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:58 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:58.809+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:53:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:59.199+0000 7f637586e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-16T22:53:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:59.199+0000 7f637586e700 -1 osd.1 46 *** Got signal Terminated *** 2024-02-16T22:53:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:59.199+0000 7f637586e700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-16T22:53:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:59.273+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:59.273+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:53:59.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:53:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:53:59.273+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:00.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:59.786+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:00.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:59.786+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:00.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:53:59 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:53:59.786+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:00.724 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:00.300+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:00.300+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:00.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:00.300+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:01.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:00.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:01.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:00.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:01.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:00 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:00.776+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:01.724 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:01.255+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:01.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:01.255+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:01.725 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:01.255+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:02.212 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:01.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:02.212 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:01.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:02.212 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:01 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:01.792+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:02.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:02.211+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:02.211+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:02.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:02.211+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:03.199 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:02.830+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:03.199 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:02.830+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:03.200 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:02 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:02.830+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:03.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:03.198+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:03.198+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:03.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:03.198+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:04.177 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:03.789+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:04.177 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:03.789+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:04.177 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:03 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:03.789+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:04.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:04.176+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:04.176+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:04.475 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1[55292]: 2024-02-16T22:54:04.176+0000 7f636ee66700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:16.592471+0000 (oldest deadline 2024-02-16T22:53:36.592471+0000) 2024-02-16T22:54:04.842 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:04 smithi002 podman[85625]: 2024-02-16 22:54:04.569141884 +0000 UTC m=+5.473694695 container died 58697cab97b069ffc618573095133a997d43d36d3fa60d1fc5dfd0b92f8ddc23 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1, RELEASE=quincy-b88fae6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-16T22:54:04.843 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:04.794+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:04.843 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:04.794+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:04.843 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:04 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:04.794+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:05.193 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:04 smithi002 podman[85625]: 2024-02-16 22:54:04.949668642 +0000 UTC m=+5.854221457 container cleanup 58697cab97b069ffc618573095133a997d43d36d3fa60d1fc5dfd0b92f8ddc23 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1, org.label-schema.build-date=20240214, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-b88fae6, org.label-schema.license=GPLv2) 2024-02-16T22:54:05.193 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:04 smithi002 bash[85625]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1 2024-02-16T22:54:05.474 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:05 smithi002 podman[85644]: 2024-02-16 22:54:05.192678676 +0000 UTC m=+0.618517507 container remove 58697cab97b069ffc618573095133a997d43d36d3fa60d1fc5dfd0b92f8ddc23 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, RELEASE=quincy-b88fae6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-16T22:54:05.821 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:05 smithi002 podman[85738]: 2024-02-16 22:54:05.406294327 +0000 UTC m=+0.018851216 image pull 870a0944ccb83fae8829e20e516a8aeec00e19c6a392cc9e32528216b9cb93f0 quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c 2024-02-16T22:54:05.821 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:05 smithi002 podman[85738]: 2024-02-16 22:54:05.526760888 +0000 UTC m=+0.139317778 container create 4f873d5d46b5b90d213e5d8f2dd80eab059154eff677ad4ec26d6299763f66ae (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=quincy-b88fae6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-16T22:54:05.821 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:05 smithi002 podman[85738]: 2024-02-16 22:54:05.737655633 +0000 UTC m=+0.350212522 container init 4f873d5d46b5b90d213e5d8f2dd80eab059154eff677ad4ec26d6299763f66ae (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-b88fae6, io.buildah.version=1.33.5, org.label-schema.build-date=20240214) 2024-02-16T22:54:05.821 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:05 smithi002 podman[85738]: 2024-02-16 22:54:05.740720643 +0000 UTC m=+0.353277537 container start 4f873d5d46b5b90d213e5d8f2dd80eab059154eff677ad4ec26d6299763f66ae (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-b88fae6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-16T22:54:06.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:05.820+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:06.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:05.820+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:06.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:05 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:05.820+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:06.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:05 smithi002 podman[85738]: 2024-02-16 22:54:05.847418915 +0000 UTC m=+0.459975810 container attach 4f873d5d46b5b90d213e5d8f2dd80eab059154eff677ad4ec26d6299763f66ae (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1-deactivate, RELEASE=quincy-b88fae6, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-16T22:54:06.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:06 smithi002 podman[85738]: 2024-02-16 22:54:06.027277752 +0000 UTC m=+0.639834648 container died 4f873d5d46b5b90d213e5d8f2dd80eab059154eff677ad4ec26d6299763f66ae (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=quincy-b88fae6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) 2024-02-16T22:54:06.974 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:06 smithi002 podman[85764]: 2024-02-16 22:54:06.657760983 +0000 UTC m=+0.622041172 container remove 4f873d5d46b5b90d213e5d8f2dd80eab059154eff677ad4ec26d6299763f66ae (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-b88fae6, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-16T22:54:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:06 smithi002 systemd[1]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-16T22:54:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:06 smithi002 systemd[1]: Stopped Ceph osd.1 for 43c11c36-cd18-11ee-95ba-87774f69a715. 2024-02-16T22:54:06.975 INFO:journalctl@ceph.osd.1.smithi002.stdout:Feb 16 22:54:06 smithi002 systemd[1]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.1.service: Consumed 17.555s CPU time. 2024-02-16T22:54:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:06.861+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:06.861+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:06.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:06 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:06.861+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:06.976 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:06 smithi002 systemd[1]: Stopping Ceph osd.2 for 43c11c36-cd18-11ee-95ba-87774f69a715... 2024-02-16T22:54:07.474 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:07.118+0000 7f2695eec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-16T22:54:07.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:07.118+0000 7f2695eec700 -1 osd.2 46 *** Got signal Terminated *** 2024-02-16T22:54:07.475 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:07.118+0000 7f2695eec700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-16T22:54:08.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:07.829+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:08.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:07.829+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:08.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:07 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:07.829+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:09.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:08.806+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:09.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:08.806+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:09.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:08 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:08.806+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:10.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:09.774+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:10.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:09.775+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:10.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:09 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:09.775+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:11.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:10.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:11.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:10.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:11.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:10 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:10.797+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:12.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:11.801+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6804 osd.3 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:12.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:11.801+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:12.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:11 smithi002 ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2[62946]: 2024-02-16T22:54:11.801+0000 7f2691ce9700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.103:6820 osd.5 ever on either front or back, first ping sent 2024-02-16T22:53:20.034827+0000 (oldest deadline 2024-02-16T22:53:40.034827+0000) 2024-02-16T22:54:12.835 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:12 smithi002 podman[85842]: 2024-02-16 22:54:12.538372242 +0000 UTC m=+5.515979495 container died 634798c2dbad54b010a3427d57d9cd7184cac6f7163ebf293194b24d48b22b35 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, RELEASE=quincy-b88fae6, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) 2024-02-16T22:54:12.836 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:12 smithi002 podman[85842]: 2024-02-16 22:54:12.835178604 +0000 UTC m=+5.812785866 container cleanup 634798c2dbad54b010a3427d57d9cd7184cac6f7163ebf293194b24d48b22b35 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-b88fae6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2) 2024-02-16T22:54:13.224 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:12 smithi002 bash[85842]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2 2024-02-16T22:54:13.225 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:13 smithi002 podman[85862]: 2024-02-16 22:54:13.061361289 +0000 UTC m=+0.516097456 container remove 634798c2dbad54b010a3427d57d9cd7184cac6f7163ebf293194b24d48b22b35 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-b88fae6, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=) 2024-02-16T22:54:13.690 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:13 smithi002 podman[85951]: 2024-02-16 22:54:13.280587815 +0000 UTC m=+0.018206296 image pull 870a0944ccb83fae8829e20e516a8aeec00e19c6a392cc9e32528216b9cb93f0 quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c 2024-02-16T22:54:13.690 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:13 smithi002 podman[85951]: 2024-02-16 22:54:13.379011769 +0000 UTC m=+0.116630244 container create 80254fd999abe5f0654061c467560cddd822ed797aa61f2da3af84b638459c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-b88fae6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-16T22:54:13.690 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:13 smithi002 podman[85951]: 2024-02-16 22:54:13.598524185 +0000 UTC m=+0.336142662 container init 80254fd999abe5f0654061c467560cddd822ed797aa61f2da3af84b638459c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=quincy-b88fae6, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-02-16T22:54:13.691 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:13 smithi002 podman[85951]: 2024-02-16 22:54:13.601616121 +0000 UTC m=+0.339234596 container start 80254fd999abe5f0654061c467560cddd822ed797aa61f2da3af84b638459c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, RELEASE=quincy-b88fae6, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-02-16T22:54:13.691 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:13 smithi002 podman[85951]: 2024-02-16 22:54:13.690059772 +0000 UTC m=+0.427678263 container attach 80254fd999abe5f0654061c467560cddd822ed797aa61f2da3af84b638459c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-b88fae6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-02-16T22:54:13.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:13 smithi002 podman[85951]: 2024-02-16 22:54:13.887190177 +0000 UTC m=+0.624808644 container died 80254fd999abe5f0654061c467560cddd822ed797aa61f2da3af84b638459c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-b88fae6, ceph=True, org.label-schema.build-date=20240214) 2024-02-16T22:54:14.586 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:14 smithi002 podman[85977]: 2024-02-16 22:54:14.559996635 +0000 UTC m=+0.667720822 container remove 80254fd999abe5f0654061c467560cddd822ed797aa61f2da3af84b638459c89 (image=quay.ceph.io/ceph-ci/ceph@sha256:bd04e78e52e49fa75ab32885b8568e6cedc3b0022775783fd0d608060059f38c, name=ceph-43c11c36-cd18-11ee-95ba-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-b88fae6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-02-16T22:54:14.974 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:14 smithi002 systemd[1]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-16T22:54:14.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:14 smithi002 systemd[1]: Stopped Ceph osd.2 for 43c11c36-cd18-11ee-95ba-87774f69a715. 2024-02-16T22:54:14.975 INFO:journalctl@ceph.osd.2.smithi002.stdout:Feb 16 22:54:14 smithi002 systemd[1]: ceph-43c11c36-cd18-11ee-95ba-87774f69a715@osd.2.service: Consumed 16.255s CPU time. 2024-02-16T22:54:18.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:54:18.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:54:21.980 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:54:30.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:54:30.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:54:34.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:54:46.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:54:46.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:54:49.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:54:52.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:54:52.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:54:52.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:54:52.188 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:54:52.188 INFO:tasks.cephadm:Teardown complete 2024-02-16T22:54:52.188 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi103 2024-02-16T22:54:52.208 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-16T22:54:52.219 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-16T22:54:52.234 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-16T22:54:52.234 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-16T22:54:52.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:54:52.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:54:55.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:55:04.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:55:04.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:55:07.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:55:19.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:55:19.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:55:22.460 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:55:25.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:55:25.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:55:25.532 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:55:25.532 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:55:25.571 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi103 2024-02-16T22:55:25.588 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-16T22:55:25.599 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-16T22:55:25.600 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-16T22:55:25.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:55:25.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:55:25.623 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-02-16T22:55:25.627 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-16T22:55:25.628 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-02-16T22:55:25.628 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 -906us[ -906us] +/- 116ms 2024-02-16T22:55:25.628 INFO:teuthology.orchestra.run.smithi002.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -1194us[-1197us] +/- 72ms 2024-02-16T22:55:25.628 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +2411us[+2408us] +/- 99ms 2024-02-16T22:55:25.628 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-16T22:55:28.672 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:55:37.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:55:37.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:55:40.736 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:55:52.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:55:52.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:55:55.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:55:58.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:55:58.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:55:58.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:55:58.876 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:55:58.877 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi103 2024-02-16T22:55:58.892 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-16T22:55:58.904 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-16T22:55:58.905 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-16T22:55:58.946 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-16T22:55:59.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:55:59.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:56:02.112 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:56:11.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:56:11.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:56:14.176 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:56:26.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:56:26.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:56:29.248 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:56:32.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:56:32.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:56:32.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:56:32.316 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:56:32.317 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi103 2024-02-16T22:56:32.325 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-16T22:56:32.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-16T22:56:32.383 INFO:teuthology.task.internal:Duration was 3138.453619 seconds 2024-02-16T22:56:32.383 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-16T22:56:32.417 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-16T22:56:32.417 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-16T22:56:32.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:56:32.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:56:32.470 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-16T22:56:35.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:56:44.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:56:44.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:56:47.548 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:56:59.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:56:59.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:57:02.620 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:57:05.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:57:05.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:57:05.696 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:57:05.696 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:57:05.697 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi103 2024-02-16T22:57:05.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-16T22:57:05.724 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-16T22:57:05.725 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-16T22:57:05.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:57:05.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:57:08.832 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:57:17.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:57:17.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:57:20.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:57:32.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:57:32.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:57:35.964 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:57:38.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:57:38.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:57:39.040 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:57:39.040 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:57:39.041 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi103 2024-02-16T22:57:39.059 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-16T22:57:39.075 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-16T22:57:39.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:57:39.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:57:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2024-02-16T22:57:42.144 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:57:51.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:57:51.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:57:54.204 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:58:06.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:58:06.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:58:09.280 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:58:12.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:58:12.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:58:12.352 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:58:12.352 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:58:12.353 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi103 2024-02-16T22:58:12.369 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-16T22:58:12.381 INFO:teuthology.task.internal:Transferring archived files... 2024-02-16T22:58:12.382 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728/remote/smithi002 2024-02-16T22:58:12.382 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-16T22:58:12.424 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728/remote/smithi103 2024-02-16T22:58:12.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:58:12.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:58:15.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:58:24.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:58:24.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:58:27.548 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:58:39.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:58:39.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:58:42.620 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:58:45.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:58:45.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:58:45.696 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:58:45.696 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:58:45.697 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi103 2024-02-16T22:58:45.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-16T22:58:45.729 INFO:teuthology.task.internal:Not uploading archives. 2024-02-16T22:58:45.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-16T22:58:45.760 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-16T22:58:45.761 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-16T22:58:45.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:58:45.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:58:45.783 INFO:teuthology.orchestra.run.smithi002.stdout: 265825 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 16 22:54 /home/ubuntu/cephtest 2024-02-16T22:58:45.783 INFO:teuthology.orchestra.run.smithi002.stdout: 265835 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 16 22:57 /home/ubuntu/cephtest/archive 2024-02-16T22:58:45.783 INFO:teuthology.orchestra.run.smithi002.stdout: 265837 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 16 22:04 /home/ubuntu/cephtest/archive/syslog 2024-02-16T22:58:45.784 INFO:teuthology.orchestra.run.smithi002.stdout: 265897 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 16 22:04 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-16T22:58:45.784 INFO:teuthology.orchestra.run.smithi002.stdout: 265848 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 16 22:04 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-16T22:58:45.784 INFO:teuthology.orchestra.run.smithi002.stdout: 402314 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 16 22:55 /home/ubuntu/cephtest/archive/audit 2024-02-16T22:58:45.784 INFO:teuthology.orchestra.run.smithi002.stdout: 426286 176 -rw------- 1 ubuntu ubuntu 179872 Feb 16 22:55 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-16T22:58:45.784 INFO:teuthology.orchestra.run.smithi002.stdout: 265374 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Feb 16 22:10 /home/ubuntu/cephtest/cephadm 2024-02-16T22:58:45.785 INFO:teuthology.orchestra.run.smithi002.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-16T22:58:48.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:58:57.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:58:57.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:59:00.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:59:12.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:59:12.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:59:15.967 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:59:18.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-02-16T22:59:18.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-16T22:59:19.036 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-02-16T22:59:19.036 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-16T22:59:19.037 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_ed7ab725e1a43615009dfb74a42bf33d80081bc8/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi103 2024-02-16T22:59:19.057 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-16T22:59:19.069 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-16T22:59:19.105 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-02-16T22:59:19.106 INFO:teuthology.nuke:Checking targets against current locks 2024-02-16T22:59:19.136 DEBUG:teuthology.nuke:shortname: smithi002 2024-02-16T22:59:19.137 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-16T22:59:19.163 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-16 21:55:18.217325', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRbrMcmK/Fgj1sNYHecH4R601mYPfWJcYpCFgAAMGA666vdznFRd/r8YlTThq7pFdB15gSrgdBTUFpbfpr8TKs='} 2024-02-16T22:59:19.165 DEBUG:teuthology.nuke:shortname: smithi103 2024-02-16T22:59:19.165 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-16T22:59:19.194 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-16_18:26:20-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7562728', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-16 21:55:18.219281', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLvBv5Qb2+qoNasbzTBfm9CW43nFJ3AuTGtkd9UtTxA6WnL9GCNIwXZoZuiZ+0OSKY6vCiHHRNCdmkPaC+xBKGs='} 2024-02-16T22:59:19.224 INFO:teuthology.orchestra.console.smithi002:Power off 2024-02-16T22:59:19.224 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-16T22:59:19.250 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2024-02-16T22:59:19.250 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-16T22:59:19.269 INFO:teuthology.orchestra.console.smithi103:Power off 2024-02-16T22:59:19.270 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-16T22:59:19.279 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-02-16T22:59:19.292 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-02-16T22:59:19.292 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-16T22:59:19.315 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-02-16T22:59:23.281 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-16T22:59:23.316 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-16T22:59:23.392 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-02-16T22:59:23.426 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-02-16T22:59:27.396 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-16T22:59:27.427 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-16T22:59:27.505 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2024-02-16T22:59:27.537 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-02-16T22:59:27.606 INFO:teuthology.orchestra.console.smithi002:Power off completed 2024-02-16T22:59:31.540 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-16T22:59:31.650 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-02-16T22:59:31.751 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-02-16T22:59:31.856 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3138.453619003296 failure_reason: 'Command failed on smithi002 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43c11c36-cd18-11ee-95ba-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=70403733bcde4eaab6618ab5a10968e1 status: fail success: false 2024-02-16T22:59:31.857 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-16T22:59:31.927 INFO:teuthology.run:FAIL